GithubHelp home page GithubHelp logo

go's Introduction

This repos contains Platina System's open source GO projects.

First fetch and install the goes builder,

$ go get github.com/platinasystems/go/main/goes-build

You'll need platinasystems/fe1 and platinasystems/firmware-fe1a to build goes-platina-mk1.

$ git clone [email protected]:platinasystems/fe1 \
	GOPATH/src/github.com/platinasystems/fe1
$ git clone [email protected]:platinasystems/firmware-fe1a 
	GOPATH/src/github.com/platinasystems/firmware-fe1a

Alternatively, you may build goes-platina-mk1 to plugin an existing /usr/lib/goes/fe1.so.

To install a select MACHINE,

$ goes-build goes-MACHINE
$ sudo ./goes-MACHINE install

Some machines also have a self extracting, compressed archive installer.

$ goes-build goes-MACHINE-installer
$ sudo ./goes-MACHINE-installer

These are the available machines,

To stop and remove,

$ sudo goes uninstall

To enable BASH completion after install,

. /usr/share/bash-completion/completions/goes

To see the commands available on the installed MACHINE,

$ goes help

Or,

$ goes
goes> help

Then man any of the listed commands or man cli to see how to use the command line interface.

See also errata.

© 2015-2016 Platina Systems, Inc. All rights reserved. Use of this source code is governed by this BSD-style LICENSE.

go's People

Contributors

1arios avatar abates avatar alialaee avatar arthurxiong3 avatar dlobete avatar donnlee avatar edressel avatar elpinchetopo avatar flisky avatar frou avatar fszyang avatar glycerine avatar gopal-platina avatar jasonlpang avatar jnjackins avatar jsternberg avatar kph avatar mattgeddes-platina avatar michaelmacinnis avatar mpleso avatar peterh avatar pires avatar rif avatar robert-coulson avatar rondv avatar sbinet avatar sharadmehrotra2 avatar stigt avatar tgrennan avatar tzneal avatar

Stargazers

avatar avatar avatar avatar avatar avatar avatar avatar avatar

Watchers

avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar

Forkers

ajeet17181 rakesh-calsoft edressel mj073 nilesh20 dlobete setiaparul55 atmecs123 sans2864 amit-pandia kph mpleso kgkannan manigadhi1 ad9195 sekhar4536 portalgun-io jefferyq sharadmehrotra2

go's Issues

FIB race condition: Can add routes to RIB before FIB is ready

I discovered that I can add rib routes quickly after 'goes start', and these routes will never make it to the fib. I think it is because linux installs the routes immediately, but fib is not ready yet (maybe because it requires next-hop /32 but linux does not?).

I wrote a script that repros the fail:

donn@invader8:~/workspace/gitlab.com/donnlee/tor/invader/testing/fib_testing$ cat quick_add_routes.py
#!/bin/bash

set -x -e

goes stop
goes start

sleep 6

/sbin/ifdown eth-$1-0
/sbin/ifup eth-$1-0

/usr/bin/goes vnet set ha sp eth-$1-0 auto
/usr/bin/goes vnet set ha sp eth-$1-0 100g
/usr/bin/goes vnet set ha sp eth-$1-0 auto dis
/usr/bin/goes vnet set ha sp eth-$1-0 auto ena

sleep 1

#for i in {1..255};
for i in `seq 0 2 255`;
  do ip ro add 20.0.0.$i/31 via 192.168.155.2;
  #do echo "20.0.0.$i/31 via 192.168.155.2";
done

The next-hop is 192.168.155.2 which is a neighboring router on this invader's eth-31-0.

After 'goes start', eth-31-0 is in the following state:

donn@invader8:~/workspace/gitlab.com/donnlee/tor/invader/testing/fib_testing$ goes vnet sh fe1 po m | ta
il
eth-24-0  false       false        false
eth-25-0  false       false        false
eth-26-0  false       false        false
eth-27-0  false       false        false
eth-28-0  false       false        false
eth-29-0  false       false        false
eth-30-0  false       false        false
eth-31-0  false        true        false

To get the link in "true true true", I do the autoneg "dance" at the top of the script.

/usr/bin/goes vnet set ha sp eth-$1-0 auto
/usr/bin/goes vnet set ha sp eth-$1-0 100g
/usr/bin/goes vnet set ha sp eth-$1-0 auto dis
/usr/bin/goes vnet set ha sp eth-$1-0 auto ena

Where $1 is "31" because I run the repro script like so:

donn@invader8:~/workspace/gitlab.com/donnlee/tor/invader/testing/fib_testing$ sudo ./quick_add_routes.py  31

The script installs ~128 static routes. These routes are installed in rib:

donn@invader8:~/workspace/gitlab.com/donnlee/tor/invader/testing/fib_testing$ ip ro
default via 192.168.101.2 dev eth0
10.2.0.0/16 dev flannel.1  proto kernel  scope link  src 10.2.65.0
10.2.65.0/24 dev docker0  proto kernel  scope link  src 10.2.65.1
20.0.0.0/31 via 192.168.155.2 dev eth-31-0
20.0.0.2/31 via 192.168.155.2 dev eth-31-0
20.0.0.4/31 via 192.168.155.2 dev eth-31-0
20.0.0.6/31 via 192.168.155.2 dev eth-31-0
20.0.0.8/31 via 192.168.155.2 dev eth-31-0
20.0.0.10/31 via 192.168.155.2 dev eth-31-0
20.0.0.12/31 via 192.168.155.2 dev eth-31-0
20.0.0.14/31 via 192.168.155.2 dev eth-31-0
20.0.0.16/31 via 192.168.155.2 dev eth-31-0
20.0.0.18/31 via 192.168.155.2 dev eth-31-0
20.0.0.20/31 via 192.168.155.2 dev eth-31-0
<...>
20.0.0.242/31 via 192.168.155.2 dev eth-31-0
20.0.0.244/31 via 192.168.155.2 dev eth-31-0
20.0.0.246/31 via 192.168.155.2 dev eth-31-0
20.0.0.248/31 via 192.168.155.2 dev eth-31-0
20.0.0.250/31 via 192.168.155.2 dev eth-31-0
20.0.0.252/31 via 192.168.155.2 dev eth-31-0
20.0.0.254/31 via 192.168.155.2 dev eth-31-0
172.17.0.0/16 dev br-131bf7797fb8  proto kernel  scope link  src 172.17.0.1
172.19.0.0/16 dev br-e7353579d7a2  proto kernel  scope link  src 172.19.0.1
192.168.101.0/24 dev eth0  proto kernel  scope link  src 192.168.101.128
192.168.155.0/24 dev eth-31-0  proto kernel  scope link  src 192.168.155.3
donn@invader8:~/workspace/gitlab.com/donnlee/tor/invader/testing/fib_testing$

BUT never make it to the fib.

donn@invader8:~/workspace/gitlab.com/donnlee/tor/invader/testing/fib_testing$ goes vnet sh ip fib
 Table                   Destination           Adjacency
     0              192.168.155.0/24  1: glean eth-31-0
     0              192.168.155.3/32  2: local eth-31-0
donn@invader8:~/workspace/gitlab.com/donnlee/tor/invader/testing/fib_testing$

MOREOVER when i manually install a route:

donn@invader8:~/workspace/gitlab.com/donnlee/tor/invader/testing/fib_testing$ sudo ip ro add 30.0.0.240/31 via 192.168.155.2

The new route is not added to fib:

donn@invader8:~/workspace/gitlab.com/donnlee/tor/invader/testing/fib_testing$ goes vnet sh ip fib
 Table                   Destination           Adjacency
     0              192.168.155.0/24  1: glean eth-31-0
     0              192.168.155.3/32  2: local eth-31-0
donn@invader8:~/workspace/gitlab.com/donnlee/tor/invader/testing/fib_testing$

Then I add a 2nd 30.x.x.x route:

donn@invader8:~/workspace/gitlab.com/donnlee/tor/invader/testing/fib_testing$ sudo ip ro add 30.0.0.242/31 via 192.168.155.2

The 2nd 30.x.x.x route appears in fib:

donn@invader8:~/workspace/gitlab.com/donnlee/tor/invader/testing/fib_testing$ goes vnet sh ip fib
 Table                   Destination           Adjacency
     0                   20.0.0.0/31  4: rewrite eth-31-0 IP4: 02:46:8a:00:05:b4 -> 00:05:64:2f:3c:c9
     0                 30.0.0.242/31  4: rewrite eth-31-0 IP4: 02:46:8a:00:05:b4 -> 00:05:64:2f:3c:c9
     0              192.168.155.0/24  1: glean eth-31-0
     0              192.168.155.2/32  3: rewrite eth-31-0 IP4: 02:46:8a:00:05:b4 -> 00:05:64:2f:3c:c9
     0              192.168.155.3/32  2: local eth-31-0
donn@invader8:~/workspace/gitlab.com/donnlee/tor/invader/testing/fib_testing$

And now, we see 192.168.155.2/32 for the first time.

The 1st 30.x.x.x route is indeed in the rib:

donn@invader8:~/workspace/gitlab.com/donnlee/tor/invader/testing/fib_testing$ ip ro | grep "30.0.0.240"
30.0.0.240/31 via 192.168.155.2 dev eth-31-0

But not in the fib:

donn@invader8:~/workspace/gitlab.com/donnlee/tor/invader/testing/fib_testing$ goes vnet sh ip fib | grep "30.0.0.240"
donn@invader8:~/workspace/gitlab.com/donnlee/tor/invader/testing/fib_testing$ goes vnet sh ip fib | grep "30.0.0.242"
     0                 30.0.0.242/31  4: rewrite eth-31-0 IP4: 02:46:8a:00:05:b4 -> 00:05:64:2f:3c:c9
donn@invader8:~/workspace/gitlab.com/donnlee/tor/invader/testing/fib_testing$

It feels like "the pump needs to be primed" somewhere, before the routes are installed in the fib.

Maybe vnet requires NH 192.168.155.2/32 but linux does not.

able to hset of sub-lane speed to "auto"; should not allow

Normally we don't allow sub-lane of an interface to be set separately. Redis seem to allow this when sub-lane speed is set to "auto". Doesn't seem to do anything adversely to the port, but behavior is incorrect.

fyang@invader17:$ goes hget platina vnet.eth-32-1.speed
100g
fyang@invader17:$ goes hset platina vnet.eth-32-2.speed 100g
hset: ERROR invalid speed
fyang@invader17:$ goes hset platina vnet.eth-32-2.speed auto
1
fyang@invader17:$ goes hset platina vnet.eth-32-3.speed auto
1
fyang@invader17:$ goes hset platina vnet.eth-32-4.speed auto
1
fyang@invader17:$ goes hget platina vnet.eth-32-2.speed
auto

mmu port flush timeout

This issue looks to be there still with latest commits from 4/27
Version: 74dfc8b
(also included commit c7d5054515784cb33860ef15aa5e5f4d17fdd4ea in fe1)

Setup
( pkgen )
| | | | | | | |
e01 e03 e05 e07 e09 e11 e13 e15 ( invader 7 )
e00 e02 e04 e06 e08 e10 e12 e14 e16 e18 e20 e22 e24 e26 e28 e30
| | | | | | | | | | | | | | | |
e00 e02 e04 e06 e08 e10 e12 e14 e16 e18 e20 e22 e24 e26 e28 e30
( invader 2 ) e17 e19 e21 e23 e25 e27 e29 e31
| | | | | | | |
( pkgen )

bidirectional traffic going from between the pkgen through invader2 and invader7

Step to reproduce:
Restart goes on invader2 (all ports on invader2 will flap)
Gets panic on invader7

Apr 28 13:01:57 invader7 goes.vnetd[1585]: mmu port flush timeout: eth-14-0: goroutine 199 [running]:
Apr 28 13:01:57 invader7 goes.vnetd[1585]: runtime/debug.Stack(0xc420035c50, 0xa54b20, 0xc42023a350)
Apr 28 13:01:57 invader7 goes.vnetd[1585]: /usr/local/go/src/runtime/debug/stack.go:24 +0x79
Apr 28 13:01:57 invader7 goes.vnetd[1585]: github.com/platinasystems/go/elib/loop.(*Loop).doEvent.func1(0xc420194000)
Apr 28 13:01:57 invader7 goes.vnetd[1585]: /home/fyang/gopath/src/github.com/platinasystems/go/elib/loop/event.go:128 +0x91
Apr 28 13:01:57 invader7 goes.vnetd[1585]: panic(0xa54b20, 0xc42023a350)
Apr 28 13:01:57 invader7 goes.vnetd[1585]: /usr/local/go/src/runtime/panic.go:458 +0x243
Apr 28 13:01:57 invader7 goes.vnetd[1585]: github.com/platinasystems/go/vnet/devices/ethernet/switch/fe1/internal/fe1a.(*Port).mmu_port_flush(0xc4203b9c70, 0xc42026e000)
Apr 28 13:01:57 invader7 goes.vnetd[1585]: /home/fyang/gopath/src/github.com/platinasystems/go/vnet/devices/ethernet/switch/fe1/internal/fe1a/mmu.go:390 +0x246
Apr 28 13:01:57 invader7 goes.vnetd[1585]: github.com/platinasystems/go/vnet/devices/ethernet/switch/fe1/internal/fe1a.(*fe1a).LinkStatusChange(0xc42026e000, 0xc420b06d28)
Apr 28 13:01:57 invader7 goes.vnetd[1585]: /home/fyang/gopath/src/github.com/platinasystems/go/vnet/devices/ethernet/switch/fe1/internal/fe1a/port_init.go:591 +0x510
Apr 28 13:01:57 invader7 goes.vnetd[1585]: github.com/platinasystems/go/vnet/devices/ethernet/switch/fe1/internal/cpu.(*linkStatusEvent).EventAction(0xc420b06d20)
Apr 28 13:01:57 invader7 goes.vnetd[1585]: /home/fyang/gopath/src/github.com/platinasystems/go/vnet/devices/ethernet/switch/fe1/internal/cpu/link_scan.go:239 +0x48
Apr 28 13:01:57 invader7 goes.vnetd[1585]: github.com/platinasystems/go/elib/loop.(*loopEvent).do(0xc420bd3320)
Apr 28 13:01:57 invader7 goes.vnetd[1585]: /home/fyang/gopath/src/github.com/platinasystems/go/elib/loop/event.go:118 +0x34
Apr 28 13:01:57 invader7 goes.vnetd[1585]: github.com/platinasystems/go/elib/loop.(*Loop).doEvent(0xc420194000, 0xc420bd3320)
Apr 28 13:01:57 invader7 goes.vnetd[1585]: /home/fyang/gopath/src/github.com/platinasystems/go/elib/loop/event.go:133 +0x51
Apr 28 13:01:57 invader7 goes.vnetd[1585]: github.com/platinasystems/go/elib/loop.(*Loop).eventHandler(0xc420194000, 0x12d7380, 0x12f6500)
Apr 28 13:01:57 invader7 goes.vnetd[1585]: /home/fyang/gopath/src/github.com/platinasystems/go/elib/loop/event.go:140 +0x86
Apr 28 13:01:57 invader7 goes.vnetd[1585]: created by github.com/platinasystems/go/elib/loop.(*Loop).startHandler
Apr 28 13:01:57 invader7 goes.vnetd[1585]: /home/fyang/gopath/src/github.com/platinasystems/go/elib/loop/event.go:153 +0x124
Apr 28 13:01:57 invader7 goes.vnetd[1585]: done

index out of range: go/vnet/ip4.showIpFib

i8 had a gobgp session established with another gobgp router (pktgen/i14).
i14 sent 4K bgp routes to i8.
Not all routes were installed.
Then i did 'vnet show ip fib' and got a panic:

donn@invader8:~/workspace/gitlab.com/donnlee/tor/invader/testing/punt_path$ gobgp neighbor
Peer             AS  Up/Down State       |#Advertised Received Accepted
10.0.29.14    65201 00:00:26 Establ      |          8      425      425
192.168.155.2 65202 00:00:29 Establ      |        431        0        0
donn@invader8:~/workspace/gitlab.com/donnlee/tor/invader/testing/punt_path$ gobgp neighbor
Peer             AS  Up/Down State       |#Advertised Received Accepted
10.0.29.14    65201 00:00:28 Establ      |          8      425      425
192.168.155.2 65202 00:00:31 Establ      |        431        0        0
donn@invader8:~/workspace/gitlab.com/donnlee/tor/invader/testing/punt_path$ goes vnet sho ip fib
 Table                   Destination           Adjacency
show ip fib: runtime error: index out of range `': goroutine 51 [running]:
runtime/debug.Stack(0xc4201b7bc0, 0xb79671, 0xb)
        /usr/local/go/src/runtime/debug/stack.go:24 +0x79
github.com/platinasystems/go/elib/cli.(*Main).ExecInput.func1(0xc420032df8, 0x12875a0, 0xc4201b7bc0, 0
xc42019a270)
        /home/dlobete/src/github.com/platinasystems/go/elib/cli/cli.go:219 +0x9c
panic(0xa79ce0, 0xc42000c0a0)
        /usr/local/go/src/runtime/panic.go:458 +0x243
github.com/platinasystems/go/vnet/devices/ethernet/switch/fe1/internal/fe1a.(*fe1a).adj_get_counter(0x
c420224000, 0xc420196768, 0x2, 0xc4211af4a0)
        /home/dlobete/src/github.com/platinasystems/go/vnet/devices/ethernet/switch/fe1/internal/fe1a/
fib.go:170 +0xdb
github.com/platinasystems/go/vnet/devices/ethernet/switch/fe1/internal/fe1a.(*fe1a).(github.com/platin
asystems/go/vnet/devices/ethernet/switch/fe1/internal/fe1a.adj_get_counter)-fm(0xc420196768, 0x2, 0xc4
211af4a0)
        /home/dlobete/src/github.com/platinasystems/go/vnet/devices/ethernet/switch/fe1/internal/fe1a/
fib.go:51 +0x46
github.com/platinasystems/go/vnet/ip.(*Main).CallAdjGetCounterHooks(0xc420196768, 0x2, 0xc4211af4a0)
        /home/dlobete/src/github.com/platinasystems/go/vnet/ip/adjacency.go:629 +0x78
github.com/platinasystems/go/vnet/ip.(*Main).ForeachAdjCounter(0xc420196768, 0xc400000002, 0xc4211af4a
0)
        /home/dlobete/src/github.com/platinasystems/go/vnet/ip/adjacency.go:680 +0x1a2
github.com/platinasystems/go/vnet/ip4.(*Main).showIpFib(0xc420196700, 0x12875a0, 0xc4201b7bc0, 0x7fd0e
045e938, 0xc4200c4780, 0xc42019a270, 0x42c993, 0xc420032d40)
        /home/dlobete/src/github.com/platinasystems/go/vnet/ip4/cli.go:86 +0x8b5
github.com/platinasystems/go/vnet/ip4.(*Main).(github.com/platinasystems/go/vnet/ip4.showIpFib)-fm(0x1
2875a0, 0xc4201b7bc0, 0x7fd0e045e938, 0xc4200c4780, 0xc42019a270, 0xc420032d00, 0x7fd0e045e938)
        /home/dlobete/src/github.com/platinasystems/go/vnet/ip4/cli.go:110 +0x5c
github.com/platinasystems/go/elib/cli.(*Command).CliAction(0xc4201b7bc0, 0x7fd0e045e938, 0xc4200c4780,
 0xc42019a270, 0xc4200c4780, 0xc42019a270)
        /home/dlobete/src/github.com/platinasystems/go/elib/cli/cli.go:53 +0x59
github.com/platinasystems/go/elib/cli.(*Main).ExecInput(0xc42019a270, 0x1283160, 0xc4200c4780, 0xc4201
9a270, 0x0, 0x0)
        /home/dlobete/src/github.com/platinasystems/go/elib/cli/cli.go:222 +0x134
github.com/platinasystems/go/elib/cli.(*Main).Exec(0xc42019a270, 0x1283160, 0xc4200c4780, 0x1284ea0, 0
xc423c3d940, 0xa, 0xc420476000)
        /home/dlobete/src/github.com/platinasystems/go/elib/cli/cli.go:230 +0x82
github.com/platinasystems/go/elib/cli.(*File).RxReady(0xc4200c4780, 0xc42047c748, 0xc42047c758)
        /home/dlobete/src/github.com/platinasystems/go/elib/cli/file.go:61 +0x1dc
github.com/platinasystems/go/elib/loop.(*fileEvent).EventAction(0xc4212c6700)
        /home/dlobete/src/github.com/platinasystems/go/elib/loop/cli.go:38 +0x33
github.com/platinasystems/go/elib/loop.(*loopEvent).do(0xc42110e840)
        /home/dlobete/src/github.com/platinasystems/go/elib/loop/event.go:118 +0x34
github.com/platinasystems/go/elib/loop.(*Loop).doEvent(0xc42019a000, 0xc42110e840)
        /home/dlobete/src/github.com/platinasystems/go/elib/loop/event.go:133 +0x51
github.com/platinasystems/go/elib/loop.(*Loop).eventHandler(0xc42019a000, 0x12876a0, 0xc42019a138)
        /home/dlobete/src/github.com/platinasystems/go/elib/loop/event.go:140 +0x86
created by github.com/platinasystems/go/elib/loop.(*Loop).startHandler
        /home/dlobete/src/github.com/platinasystems/go/elib/loop/event.go:153 +0x124

donn@invader8:~/workspace/gitlab.com/donnlee/tor/invader/testing/punt_path$

donn@invader8:~/workspace/gitlab.com/donnlee/tor/invader/testing/punt_path$ /usr/bin/goes version
1df40b36f56a005cb48fe61249c0513f5b440087

mmu_unicast_tx_cos0_drop on half of the ports

I have traffic flowing bi-directionally on all 32 ports using static arps. Ports eth- 0,2,4,6,8,10,12,14 are connected to traffic gen. All other 24 ports are connected to another switch as "uplink" ports. Each traffic gen ports has 24 streams of traffic going to each of these uplink ports.

If I disable any ONE port from eth-16-0 to eth-31-0, I get cos0 drop counter increment on ALL ports eth-16-0 through eth-31-0.

If I disable any ONE port from eth 1,3,5,7,9,11,13, or 15 (i.e. ports facing foxy4), I get cos0 drop counter increment on ALL ports eth-0-0 through eth-15-0.

Example:

root@invader2:/home/platina# goes hgetall platina eth- | grep link
eth-0-0.link: true
eth-1-0.link: true
eth-10-0.link: true
eth-11-0.link: true
eth-12-0.link: true
eth-13-0.link: true
eth-14-0.link: true
eth-15-0.link: true
eth-16-0.link: true
eth-17-0.link: true
eth-18-0.link: true
eth-19-0.link: true
eth-2-0.link: true
eth-20-0.link: false
eth-21-0.link: true
eth-22-0.link: true
eth-23-0.link: true
eth-24-0.link: true
eth-25-0.link: true
eth-26-0.link: true
eth-27-0.link: true
eth-28-0.link: true
eth-29-0.link: true
eth-3-0.link: true
eth-30-0.link: true
eth-31-0.link: true
eth-4-0.link: true
eth-5-0.link: true
eth-6-0.link: true
eth-7-0.link: true
eth-8-0.link: true
eth-9-0.link: true

goes hdelta -i 5 | grep cos0_drop_bytes
...

eth-16-0.mmu_unicast_tx_cos0_drop_bytes: 5.10385212e+08 (1.02e+08/s)
eth-17-0.mmu_unicast_tx_cos0_drop_bytes: 5.10315552e+08 (1.02e+08/s)
eth-18-0.mmu_unicast_tx_cos0_drop_bytes: 5.1029514e+08 (1.02e+08/s)
eth-19-0.mmu_unicast_tx_cos0_drop_bytes: 5.1031296e+08 (1.02e+08/s)
eth-20-0.mmu_unicast_tx_cos0_drop_bytes: 5.1026436e+08 (1.02e+08/s)
eth-21-0.mmu_unicast_tx_cos0_drop_bytes: 5.10417288e+08 (1.02e+08/s)
eth-22-0.mmu_unicast_tx_cos0_drop_bytes: 5.10253668e+08 (1.02e+08/s)
eth-23-0.mmu_unicast_tx_cos0_drop_bytes: 5.10413724e+08 (1.02e+08/s)
eth-24-0.mmu_unicast_tx_cos0_drop_bytes: 5.10366582e+08 (1.02e+08/s)
eth-25-0.mmu_unicast_tx_cos0_drop_bytes: 5.103486e+08 (1.02e+08/s)
eth-26-0.mmu_unicast_tx_cos0_drop_bytes: 5.10298866e+08 (1.02e+08/s)
eth-27-0.mmu_unicast_tx_cos0_drop_bytes: 5.10243786e+08 (1.02e+08/s)
eth-28-0.mmu_unicast_tx_cos0_drop_bytes: 5.10374682e+08 (1.02e+08/s)
eth-29-0.mmu_unicast_tx_cos0_drop_bytes: 5.1047091e+08 (1.02e+08/s)
eth-30-0.mmu_unicast_tx_cos0_drop_bytes: 5.10367068e+08 (1.02e+08/s)
eth-31-0.mmu_unicast_tx_cos0_drop_bytes: 5.10387156e+08 (1.02e+08/s)

MMU flush on neighbor link up

Need to flush fe1a mmu queues. Probably need a similar mechanism for vnet (for any packets sitting in tx-ring). Eliot needs to think about how to hook up in the link-down thread of execution.

interface media copper must be set before speed in /etc/goes/start

In /etc/goes/start, interface media copper config must come before interface speed config if running copper (i.e. cable). Otherwise autoneg or fixed speed will not tune the adaptive DFE and therefore result in a sub optimal received eye that could lead to crcs or link flap.

In general, media copper setting must be done before speed setting. For example manually via redis.

If not done in correct order, then issue speed setting in redis again to trigger the adaptive DFE to start again.

admin link state in redis not updating

Version: 20fd9f2
Saw this on just eth-11-0 of invader2. redis field vnet.eth-11-0.admin not reflecting actual state in Linux. Changed Linux state up and down via ip dev set several times. No change in redis admin state. Verified that netlink [link] messages are being generated.

Recovered after goes restart
Not easily reproducible

platina@invader2:$ ip link show eth-11-0
49: eth-11-0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 4352 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 5000
link/ether 02:46:8a:00:03:d2 brd ff:ff:ff:ff:ff:f

platina@invader2:$ goes hget platina eth-11-0.admin
vnet.eth-11-0.admin: false

platina@invader2:$ sudo ip link set dev eth-11-0 down
platina@invader2:~$ ip link show eth-11-0
49: eth-11-0: <BROADCAST,MULTICAST> mtu 4352 qdisc noqueue state DOWN mode DEFAULT group default qlen 5000
link/ether 02:46:8a:00:03:d2 brd ff:ff:ff:ff:ff:ff

platina@invader2:$ goes hget platina eth-11-0.admin
vnet.eth-11-0.admin: false

platina@invader2:$ sudo ip link set dev eth-11-0 up
platina@invader2:$ ip link show eth-11-0
49: eth-11-0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 4352 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 5000
link/ether 02:46:8a:00:03:d2 brd ff:ff:ff:ff:ff:ff

platina@invader2:$ goes hget platina eth-11-0.admin
vnet.eth-11-0.admin: false

Netlink message captures:
[LINK]49: eth-11-0: <BROADCAST,MULTICAST> mtu 4352 qdisc noqueue state DOWN group default
link/ether 02:46:8a:00:03:d2 brd ff:ff:ff:ff:ff:ff
[LINK]49: eth-11-0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 4352 state UNKNOWN
link/ether 02:46:8a:00:03:d2

goes doesn't understand the redis 'quit' and 'info' commands

goes throws an error when redis client sends the QUIT & INFO commands

https://redis.io/commands/quit

https://redis.io/commands/info

Some redis clients send INFO after the connection is established. These clients see an error response from goes.

The following is a tcpdump showing that goes doesn't understand QUIT:

07:45:50.464104 IP 192.168.101.128.6379 > 172.16.3.108.49034: Flags [S.], seq 3664199404, ack 2464616518, win 29200, options [mss 1460,nop,nop,sackOK,nop,wscale 8], length 0
07:45:50.464120 IP 172.16.3.108.49034 > 192.168.101.128.6379: Flags [.], ack 1, win 229, length 0
07:45:50.466959 IP 172.16.3.108.49034 > 192.168.101.128.6379: Flags [P.], seq 1:54, ack 1, win 229, length 53: RESP "hget" "platina" "eeprom.Manufacturer"
07:45:50.468102 IP 192.168.101.128.6379 > 172.16.3.108.49034: Flags [.], ack 54, win 115, length 0
07:45:50.468118 IP 192.168.101.128.6379 > 172.16.3.108.49034: Flags [P.], seq 1:5, ack 54, win 115, length 4: RESP [|RESP]
07:45:50.468123 IP 172.16.3.108.49034 > 192.168.101.128.6379: Flags [.], ack 5, win 229, length 0
07:45:50.468265 IP 192.168.101.128.6379 > 172.16.3.108.49034: Flags [P.], seq 5:14, ack 54, win 115, length 9: RESP "Platina"
07:45:50.468269 IP 172.16.3.108.49034 > 192.168.101.128.6379: Flags [.], ack 14, win 229, length 0
07:45:50.468823 IP 172.16.3.108.49034 > 192.168.101.128.6379: Flags [P.], seq 54:68, ack 14, win 229, length 14: RESP "quit"
07:45:50.469769 IP 192.168.101.128.6379 > 172.16.3.108.49034: Flags [P.], seq 14:46, ack 68, win 115, length 32: RESP "ERROR Method is not supported"
07:45:50.477889 IP 172.16.3.108.49034 > 192.168.101.128.6379: Flags [F.], seq 68, ack 46, win 229, length 0
07:45:50.478751 IP 192.168.101.128.6379 > 172.16.3.108.49034: Flags [P.], seq 46:51, ack 69, win 115, length 5: RESP [|RESP]
07:45:50.478768 IP 172.16.3.108.49034 > 192.168.101.128.6379: Flags [R], seq 2464616586, win 0, length 0
07:45:50.478801 IP 192.168.101.128.6379 > 172.16.3.108.49034: Flags [F.], seq 51, ack 69, win 115, length 0
07:45:50.478805 IP 172.16.3.108.49034 > 192.168.101.128.6379: Flags [R], seq 2464616586, win 0, length 0

panic: tuntap ioctl SETIFHWADDR: cannot assign requested address: 00:00:00:00:00:00

I'm seeing the following panic about 1 out of 10 times with "./goes-platina-mk1 install". I'm at top of tree with go and fe1:
go: commit 0bd1317
fe1: commit 87d0ed69f629bf509830e2a0be053a603dd36bab

Dec 8 18:05:09 platina systemd[1]: Reloading.
Dec 8 18:05:09 platina systemd[1]: Started ACPI event daemon.
Dec 8 18:05:09 platina systemd[1]: Listening on ACPID Listen Socket.
Dec 8 18:05:09 platina systemd[1]: Activated swap /dev/disk/by-uuid/46931bab-e3fb-4f43-be65-7575289bf444.
Dec 8 18:05:09 platina systemd[1]: Activated swap /dev/sda5.
Dec 8 18:05:09 platina systemd[1]: Activated swap /dev/disk/by-id/ata-SanDisk_SD8SMAT128G1122_162823401251-part5.
Dec 8 18:05:09 platina systemd[1]: Activated swap /dev/disk/by-id/wwn-0x5001b444a409787f-part5.
Dec 8 18:05:09 platina goes.redisd[1916]: listen: /run/goes/socks/redisd
Dec 8 18:05:09 platina goes.redisd[1916]: listen: [127.0.0.1%lo]:6379
Dec 8 18:05:09 platina goes.redisd[1916]: listen: [::1%lo]:6379
Dec 8 18:05:09 platina goes.redisd[1916]: listen: [192.168.101.213%eth0]:6379
Dec 8 18:05:09 platina goes.redisd[1916]: listen: [fe80::205:64ff:febb:bb84%eth0]:6379
Dec 8 18:05:11 platina goes.i2cd[1926]: listen OKAY
Dec 8 18:05:11 platina goes.vnetd[1932]: panic: tuntap ioctl SETIFHWADDR: cannot assign requested address: 00:00:00:00:00:00
Dec 8 18:05:11 platina goes.vnetd[1932]:
Dec 8 18:05:11 platina goes.vnetd[1932]: goroutine 1 [running]:
Dec 8 18:05:11 platina goes.vnetd[1932]: panic(0xa75680, 0xc42006e540)
Dec 8 18:05:11 platina goes.vnetd[1932]: /usr/local/go/src/runtime/panic.go:500 +0x1a1
Dec 8 18:05:11 platina goes.vnetd[1932]: github.com/platinasystems/go/vnet.(*Vnet).NewSwIf(0xc4200eab00, 0x100000001, 0xc42086d4d8)
Dec 8 18:05:11 platina goes.vnetd[1932]: /home/jpang/gopath/src/github.com/platinasystems/go/vnet/interface.go:161 +0x15e
Dec 8 18:05:11 platina goes.vnetd[1932]: github.com/platinasystems/go/vnet.(*Vnet).RegisterAndProvisionHwInterface(0xc4200eab00, 0x7f498fba47e8, 0xc4201f4500, 0x7f498fba4701, 0xb8c7fd, 0xc, 0xc42086d5c8, 0x3, 0x3, 0xc4203c54fb, ...)
Dec 8 18:05:11 platina goes.vnetd[1932]: /home/jpang/gopath/src/github.com/platinasystems/go/vnet/interface.go:399 +0x23f
Dec 8 18:05:11 platina goes.vnetd[1932]: github.com/platinasystems/go/vnet/ethernet.RegisterInterface(0xc4200eab00, 0x12ae800, 0xc4201f4500, 0xc4201f4808, 0xb8c7fd, 0xc, 0xc42086d5c8, 0x3, 0x3)
Dec 8 18:05:11 platina goes.vnetd[1932]: /home/jpang/gopath/src/github.com/platinasystems/go/vnet/ethernet/interface.go:159 +0xed
Dec 8 18:05:11 platina goes.vnetd[1932]: github.com/platinasystems/go/vnet/devices/ethernet/ixge.(*dev).vnetInit(0xc4201f4500)
Dec 8 18:05:11 platina goes.vnetd[1932]: /home/jpang/gopath/src/github.com/platinasystems/go/vnet/devices/ethernet/ixge/vnet.go:36 +0x282
Dec 8 18:05:11 platina goes.vnetd[1932]: github.com/platinasystems/go/vnet/devices/ethernet/ixge.(*dev).Init(0xc4201f4500)
Dec 8 18:05:11 platina goes.vnetd[1932]: /home/jpang/gopath/src/github.com/platinasystems/go/vnet/devices/ethernet/ixge/init.go:145 +0x278
Dec 8 18:05:11 platina goes.vnetd[1932]: github.com/platinasystems/go/elib/hw/pci.DiscoverDevices(0x0, 0x0)
Dec 8 18:05:11 platina goes.vnetd[1932]: /home/jpang/gopath/src/github.com/platinasystems/go/elib/hw/pci/linux_pci.go:199 +0x69d
Dec 8 18:05:11 platina goes.vnetd[1932]: github.com/platinasystems/go/vnet/devices/bus/pci.(*pciDiscover).Init(0xc42018ad90, 0x4, 0x6)
Dec 8 18:05:11 platina goes.vnetd[1932]: /home/jpang/gopath/src/github.com/platinasystems/go/vnet/devices/bus/pci/pci.go:14 +0x22
Dec 8 18:05:11 platina goes.vnetd[1932]: github.com/platinasystems/go/vnet.(*packageMain).InitPackages(0xc4200eb4c0, 0xc4200de540, 0x0)
Dec 8 18:05:11 platina goes.vnetd[1932]: /home/jpang/gopath/src/github.com/platinasystems/go/vnet/package.go:156 +0x30b
Dec 8 18:05:11 platina goes.vnetd[1932]: github.com/platinasystems/go/vnet.(*Vnet).configure(0xc4200eab00, 0xc4200de540, 0xbff110, 0xc4201c5a90)
Dec 8 18:05:11 platina goes.vnetd[1932]: /home/jpang/gopath/src/github.com/platinasystems/go/vnet/vnet.go:43 +0x64
Dec 8 18:05:11 platina goes.vnetd[1932]: github.com/platinasystems/go/vnet.(*Vnet).Run.func1(0xc4200eab00)
Dec 8 18:05:11 platina goes.vnetd[1932]: /home/jpang/gopath/src/github.com/platinasystems/go/vnet/vnet.go:58 +0xd1
Dec 8 18:05:11 platina goes.vnetd[1932]: github.com/platinasystems/go/elib/loop.(*Loop).callInitHooks(0xc4200eab00)
Dec 8 18:05:11 platina goes.vnetd[1932]: /home/jpang/gopath/src/github.com/platinasystems/go/elib/loop/loop.go:389 +0x5d
Dec 8 18:05:11 platina goes.vnetd[1932]: github.com/platinasystems/go/elib/loop.(*Loop).Run(0xc4200eab00)
Dec 8 18:05:11 platina goes.vnetd[1932]: /home/jpang/gopath/src/github.com/platinasystems/go/elib/loop/loop.go:441 +0xf0
Dec 8 18:05:11 platina goes.vnetd[1932]: github.com/platinasystems/go/vnet.(*Vnet).Run(0xc4200eab00, 0xc4200de540, 0x1, 0x1)
Dec 8 18:05:11 platina goes.vnetd[1932]: /home/jpang/gopath/src/github.com/platinasystems/go/vnet/vnet.go:62 +0xab
Dec 8 18:05:11 platina goes.vnetd[1932]: github.com/platinasystems/go/internal/goes/cmd/vnetd.(*cmd).Main(0xc4200eab00, 0xc42006e160, 0x0, 0x0, 0x0, 0x0)
Dec 8 18:05:11 platina goes.vnetd[1932]: /home/jpang/gopath/src/github.com/platinasystems/go/internal/goes/cmd/vnetd/vnetd.go:116 +0x689
Dec 8 18:05:11 platina goes.vnetd[1932]: github.com/platinasystems/go/internal/goes.(mainer).Main-fm(0xc42006e160, 0x0, 0x0, 0xc420158120, 0xc420045d10)
Dec 8 18:05:11 platina goes.vnetd[1932]: /home/jpang/gopath/src/github.com/platinasystems/go/internal/goes/goes.go:223 +0x4d
Dec 8 18:05:11 platina goes.vnetd[1932]: github.com/platinasystems/go/internal/goes.ByName.Main(0xc4201306c0, 0xc42006e160, 0x0, 0x0, 0x0, 0x0)
Dec 8 18:05:11 platina goes.vnetd[1932]: /home/jpang/gopath/src/github.com/platinasystems/go/internal/goes/goes.go:192 +0x588
Dec 8 18:05:11 platina goes.vnetd[1932]: main.main()
Dec 8 18:05:11 platina goes.vnetd[1932]: /home/jpang/gopath/src/github.com/platinasystems/go/main/goes-platina-mk1/main.go:14 +0x4c
Dec 8 18:05:11 platina goes.vnetd[1932]: exit status 2

panic on "goes stop"

This might be lower priority since it's happening on a goes stop and so far only reproducible on invader24 which has traffic flowing.

  • Run 1 iteration of torture script:

root@invader24:/home/stig# ./torture 1
Wed Mar 1 18:18:12 PST 2017
interation 1
stopping
starting
vnet.ready = true
reading vnet counters
val = 738
Add route - OK
Del route - OK
Done

  • Check the fib on the interface with static route added:

root@invader24:/home/stig# goes vnet show ip fib | grep eth-1-1
0 1.1.1.0/24 579: glean eth-1-1
0 1.1.1.1/32 580: local eth-1-1
0 1.1.1.2/32 581: rewrite eth-1-1 IP4: 6c:ec:5a:07:d1:a0 -> 00:01:02:03:04:05

  • Stop goes, syslog shows:

Mar 1 18:29:35 invader24 goes.vnetd[29345]: panic in addDelReplace 1.1.1.1/24 4294967294 true
Mar 1 18:29:35 invader24 goes.vnetd[29345]: addDelReplace: goroutine 36 [running]:
Mar 1 18:29:35 invader24 goes.vnetd[29345]: runtime/debug.Stack(0x0, 0x0, 0x0)
Mar 1 18:29:35 invader24 goes.vnetd[29345]: /usr/local/go/src/runtime/debug/stack.go:24 +0x80
Mar 1 18:29:35 invader24 goes.vnetd[29345]: github.com/platinasystems/go/elib/loop.(*Loop).doEvent.func1(0xc82018e000)
Mar 1 18:29:35 invader24 goes.vnetd[29345]: /home/stig/go/src/github.com/platinasystems/go/elib/loop/event.go:128 +0xa2
Mar 1 18:29:35 invader24 goes.vnetd[29345]: panic(0xbaa2e0, 0xc82012dfe0)
Mar 1 18:29:35 invader24 goes.vnetd[29345]: /usr/local/go/src/runtime/panic.go:426 +0x4e9
Mar 1 18:29:35 invader24 goes.vnetd[29345]: github.com/platinasystems/go/vnet/ip4.(*Fib).addDelReplace(0xc820472160, 0xc82006c700, 0xc82012df20, 0x1fffffffe)
Mar 1 18:29:35 invader24 goes.vnetd[29345]: /home/stig/go/src/github.com/platinasystems/go/vnet/ip4/fib.go:457 +0x4a3
Mar 1 18:29:35 invader24 goes.vnetd[29345]: github.com/platinasystems/go/vnet/ip4.(*Main).addDelInterfaceRoutes(0xc82006c700, 0x100000020)
Mar 1 18:29:35 invader24 goes.vnetd[29345]: /home/stig/go/src/github.com/platinasystems/go/vnet/ip4/fib.go:482 +0x37f
Mar 1 18:29:35 invader24 goes.vnetd[29345]: github.com/platinasystems/go/vnet/ip4.(*Main).swIfAdminUpDown.func1(0x20, 0xc820649480, 0x0, 0x0)
Mar 1 18:29:35 invader24 goes.vnetd[29345]: /home/stig/go/src/github.com/platinasystems/go/vnet/ip4/fib.go:563 +0x45
Mar 1 18:29:35 invader24 goes.vnetd[29345]: github.com/platinasystems/go/vnet/ip.(*ifAddressMain).ForeachIfAddress(0xc82006cb58, 0xc80000000d, 0xc838175c30, 0x0, 0x0)
Mar 1 18:29:35 invader24 goes.vnetd[29345]: /home/stig/go/src/github.com/platinasystems/go/vnet/ip/interface.go:74 +0xc6
Mar 1 18:29:35 invader24 goes.vnetd[29345]: github.com/platinasystems/go/vnet/ip4.(*Main).swIfAdminUpDown(0xc82006c700, 0xc82018e000, 0xd, 0x0, 0x0)
Mar 1 18:29:35 invader24 goes.vnetd[29345]: /home/stig/go/src/github.com/platinasystems/go/vnet/ip4/fib.go:565 +0x89
Mar 1 18:29:35 invader24 goes.vnetd[29345]: github.com/platinasystems/go/vnet/ip4.(*Main).(github.com/platinasystems/go/vnet/ip4.swIfAdminUpDown)-fm(0xc82018e000, 0xd, 0x0, 0x0)
Mar 1 18:29:35 invader24 goes.vnetd[29345]: /home/stig/go/src/github.com/platinasystems/go/vnet/ip4/package.go:28 +0x49
Mar 1 18:29:35 invader24 goes.vnetd[29345]: github.com/platinasystems/go/vnet.(*swIf).SetAdminUp(0xc82081c0d0, 0xc82018e000, 0x568d00, 0x0, 0x0)
Mar 1 18:29:35 invader24 goes.vnetd[29345]: /home/stig/go/src/github.com/platinasystems/go/vnet/interface.go:225 +0xce
Mar 1 18:29:35 invader24 goes.vnetd[29345]: github.com/platinasystems/go/vnet.Si.SetAdminUp(0xd, 0xc82018e000, 0x4900, 0x0, 0x0)
Mar 1 18:29:35 invader24 goes.vnetd[29345]: /home/stig/go/src/github.com/platinasystems/go/vnet/interface.go:235 +0x8a
Mar 1 18:29:35 invader24 goes.vnetd[29345]: github.com/platinasystems/go/vnet/unix.(*netlinkEvent).EventAction(0xc820a3ef90)
Mar 1 18:29:35 invader24 goes.vnetd[29345]: /home/stig/go/src/github.com/platinasystems/go/vnet/unix/netlink.go:286 +0x86d
Mar 1 18:29:35 invader24 goes.vnetd[29345]: github.com/platinasystems/go/elib/loop.(*loopEvent).do(0xc820a3efc0)
Mar 1 18:29:35 invader24 goes.vnetd[29345]: /home/stig/go/src/github.com/platinasystems/go/elib/loop/event.go:118 +0x135
Mar 1 18:29:35 invader24 goes.vnetd[29345]: github.com/platinasystems/go/elib/loop.(*Loop).doEvent(0xc82018e000, 0xc820a3efc0)
Mar 1 18:29:35 invader24 goes.vnetd[29345]: /home/stig/go/src/github.com/platinasystems/go/elib/loop/event.go:133 +0x48
Mar 1 18:29:35 invader24 goes.vnetd[29345]: github.com/platinasystems/go/elib/loop.(*Loop).eventHandler(0xc82018e000, 0x7fd622fd51d8, 0xc82018e548)
Mar 1 18:29:35 invader24 goes.vnetd[29345]: /home/stig/go/src/github.com/platinasystems/go/elib/loop/event.go:140 +0x81
Mar 1 18:29:35 invader24 goes.vnetd[29345]: created by github.com/platinasystems/go/elib/loop.(*Loop).startHandler
Mar 1 18:29:35 invader24 goes.vnetd[29345]: /home/stig/go/src/github.com/platinasystems/go/elib/loop/event.go:153 +0x120
Mar 1 18:29:35 invader24 goes.redisd[29311]: killed
Mar 1 18:29:35 invader24 goes.nld[29327]: killed
Mar 1 18:29:35 invader24 goes.qsfpio[29331]: killed
Mar 1 18:29:35 invader24 goes.uptimed[29326]: killed
Mar 1 18:29:35 invader24 goes.uptimed[29326]: done
Mar 1 18:29:35 invader24 goes.qsfp[29334]: killed
Mar 1 18:29:35 invader24 goes.qsfp[29334]: done
Mar 1 18:29:35 invader24 goes.i2cd[29354]: killed
Mar 1 18:29:35 invader24 goes.i2cd[29354]: done
Mar 1 18:29:35 invader24 goes.qsfpio[29331]: done
Mar 1 18:29:35 invader24 goes.nld[29327]: done
Mar 1 18:29:35 invader24 goes.redisd[29311]: done
Mar 1 18:29:37 invader24 goes.vnetd[29345]: done
Mar 1 18:29:37 invader24 goes.goes-daemons[29306]: done

  • Adding a print in addDelReplace show deleting the same prefix twice:

Mar 1 18:29:35 invader24 goes.vnetd[29345]: addDelReplace 1.1.1.1/24 4294967294 true
Mar 1 18:29:35 invader24 goes.vnetd[29345]: addDelReplace 1.1.1.1/32 4294967294 true
Mar 1 18:29:35 invader24 goes.vnetd[29345]: addDelReplace 1.1.1.1/24 4294967294 true
Mar 1 18:29:35 invader24 goes.vnetd[29345]: panic in addDelReplace 1.1.1.1/24 4294967294 true

CPS reset timeout (fe1a not appearing on PCI bus and system needs hard power)

Need to hard-reset/unreset of fe1a instead of soft/cps reset.
Should "goes stop" put fe1a into hard-reset ? Probably if in a panic situation but for a sw upgrade where fe1a is expected to be back up in a very short time not i.e. it would seem ok to have fe1a forward packets while headless for a short time but not indefinitely.

fib: Extra /32 is added to dummy0 after admin-up

Before goes startup, I have a dummy0 interface with 5.5.5.5/32:

donn@invader8:~/workspace/go/src/github.com/platinasystems/go$ ip add sho dummy0
9603: dummy0: <BROADCAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 10
00
    link/ether 6e:c1:22:ab:15:5b brd ff:ff:ff:ff:ff:ff
    inet 5.5.5.5/0 brd 255.255.255.255 scope global dummy0
       valid_lft forever preferred_lft forever
    inet6 fe80::6cc1:22ff:feab:155b/64 scope link
       valid_lft forever preferred_lft forever

Then I start goes:

donn@invader8:~/workspace/go/src/github.com/platinasystems/go$ sudo goes restart
donn@invader8:~/workspace/go/src/github.com/platinasystems/go$

I expect the fib to have an entry for dummy0's address, but it does not:

donn@invader8:~/workspace/go/src/github.com/platinasystems/go$ goes vnet sho ip fib
 Table                   Destination           Adjacency
     0                     0.0.0.0/0  2: punt
donn@invader8:~/workspace/go/src/github.com/platinasystems/go$

To force vnet to add a fib entry for dummy0, I had to delete and re-add dummy0, like so:

donn@invader8:~/workspace/go/src/github.com/platinasystems/go$ sudo ip link del dummy0
donn@invader8:~/workspace/go/src/github.com/platinasystems/go$ sudo ip link add dummy0 type dummy
donn@invader8:~/workspace/go/src/github.com/platinasystems/go$ sudo ifconfig dummy0 3.3.3.3/32
donn@invader8:~/workspace/go/src/github.com/platinasystems/go$
donn@invader8:~/workspace/go/src/github.com/platinasystems/go$ goes vnet sho ip fib
 Table                   Destination           Adjacency
     0                     0.0.0.0/0  2: punt
     0                    3.3.3.3/32  2: punt
donn@invader8:~/workspace/go/src/github.com/platinasystems/go$

Expected condition: I expect goes to install any dummy interface addresses it finds upon starting up.

ping doesn't work to/from front-panel ports after certain load

SW version: fe1252f

Setup:

  1. Continuous ping from a port, say eth-22-0, on invader 7 to another invader. Slow rate, 1 per second. Working.
  2. Send background bi-directional unicast traffic on 24 ports, including eth-22-0. Traffic rate is no more than 30Gbps per port.
  3. Ping stops working. All TX ping packets from invader7 dropped at its mmu_multicast_tx_cos0 queue.
  4. Stopped the background traffic so just the ping traffic left.
  5. Ping still not working. All TX ping packets from invader7 dropped at its mmu_multicast_tx_cos0 queue.

Have to restart goes to recover.

(Just blasting unicast traffic out of eth-22-0 alone does not seem to stop ping from going out eth-22-0.)

deleting tuntap interface cause vnetd to panic

Probably low priority since we're moving away from tuntap.

root@invader24:/home/stig# ip add sh eth-1-1
20411: eth-1-1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 4352 qdisc pfifo_fast state UP group default qlen 5000
link/ether 6c:ec:5a:07:d1:a0 brd ff:ff:ff:ff:ff:ff
inet 10.0.1.24/24 brd 10.0.1.255 scope global eth-1-1
valid_lft forever preferred_lft forever
inet6 fe80::6eec:5aff:fe07:d1a0/64 scope link
valid_lft forever preferred_lft forever

root@invader24:/home/stig# ip link del eth-1-1

Mar 1 21:08:02 invader24 goes.vnetd[32249]: panic: socket operation on non-socket
Mar 1 21:08:02 invader24 goes.vnetd[32249]:
Mar 1 21:08:02 invader24 goes.vnetd[32249]: goroutine 18 [running]:
Mar 1 21:08:02 invader24 goes.vnetd[32249]: panic(0xe16e40, 0xc8211faae8)
Mar 1 21:08:02 invader24 goes.vnetd[32249]: /usr/local/go/src/runtime/panic.go:464 +0x3e6
Mar 1 21:08:02 invader24 goes.vnetd[32249]: github.com/platinasystems/go/vnet/unix.(*Interface).ErrorReady(0xc82035f400, 0x7f270e4a4050, 0xc8211faae8)
Mar 1 21:08:02 invader24 goes.vnetd[32249]: /home/stig/go/src/github.com/platinasystems/go/vnet/unix/node.go:366 +0xc8
Mar 1 21:08:02 invader24 goes.vnetd[32249]: github.com/platinasystems/go/elib/iomux.(*Mux).do(0x1924500, 0xc82049b300)
Mar 1 21:08:02 invader24 goes.vnetd[32249]: /home/stig/go/src/github.com/platinasystems/go/elib/iomux/epoll.go:153 +0x329
Mar 1 21:08:02 invader24 goes.vnetd[32249]: github.com/platinasystems/go/elib/iomux.(*Mux).EventPoll(0x1924500)
Mar 1 21:08:02 invader24 goes.vnetd[32249]: /home/stig/go/src/github.com/platinasystems/go/elib/iomux/epoll.go:166 +0x24b
Mar 1 21:08:02 invader24 goes.vnetd[32249]: github.com/platinasystems/go/elib/loop.(*eventLoop).eventPoller(0xc8200d4e70, 0x7f270e3e4000, 0x1924500)
Mar 1 21:08:02 invader24 goes.vnetd[32249]: /home/stig/go/src/github.com/platinasystems/go/elib/loop/event.go:158 +0x27
Mar 1 21:08:02 invader24 goes.vnetd[32249]: created by github.com/platinasystems/go/elib/loop.(*eventLoop).startPoller
Mar 1 21:08:02 invader24 goes.vnetd[32249]: /home/stig/go/src/github.com/platinasystems/go/elib/loop/event.go:161 +0x49
Mar 1 21:08:04 invader24 goes.vnetd[32249]: exit status 2

vnet hangs

vnet hangs sometimes. stats output interval to redis starts to stretch beyond 5 seconds. hset to vnet fields like speed would time out. vnet cli command would hang

How to reproduce:
Attached bash script sets ports from 100g to auto and back and repeats. The other end of the box is connected to another switch so in the process link will flap up/down.

After about 1600 loops, hset speed commands starts to time out. vnet is hung at that point.
Attached also is the etc/goes/start file

start.txt
test_port_noeye-bash.txt

netlink listen sometimes fails on goes start

Apr 3 17:58:54 invader1 goes.redisd[9036]: listen: /run/goes/socks/redisd
Apr 3 17:58:54 invader1 goes.redisd[9036]: listen: [127.0.0.1%lo]:6379
Apr 3 17:58:54 invader1 goes.redisd[9036]: listen: [::1%lo]:6379
Apr 3 17:58:54 invader1 goes.redisd[9036]: listen: [192.168.101.206%eth0]:6379
Apr 3 17:58:54 invader1 goes.redisd[9036]: listen: [192.168.101.121%eth0]:6379
Apr 3 17:58:54 invader1 goes.redisd[9036]: listen: [fe80::46:8aff:fe00:2a0%eth0]:6379
Apr 3 17:58:56 invader1 goes.i2cd[9049]: listen OKAY
Apr 3 17:59:01 invader1 goes.i2cd[9049]: Error doing I2C R/W: bus 0x0 addr 0x50 offset 0x7f data 0x3 RW 0 BusSize 2 delay 0
Apr 3 17:59:01 invader1 goes.qsfp[9068]: i2cReq error:write: no such device or address
Apr 3 17:59:01 invader1 goes.i2cd[9049]: Error doing I2C R/W: bus 0x0 addr 0x50 offset 0x7f data 0x0 RW 0 BusSize 2 delay 0
Apr 3 17:59:01 invader1 goes.qsfp[9068]: i2cReq error:write: no such device or address
Apr 3 17:59:01 invader1 goes.qsfp[9068]: QSFP detected in port 1: Unspecified, Molex Inc., 1002971301, 526520698
Apr 3 17:59:01 invader1 goes.i2cd[9049]: Error doing I2C R/W: bus 0x0 addr 0x50 offset 0x7f data 0x0 RW 0 BusSize 2 delay 0
Apr 3 17:59:01 invader1 goes.qsfp[9068]: i2cReq error:write: no such device or address
Apr 3 17:59:01 invader1 goes.vnetd[9061]: panic: invalid argument
Apr 3 17:59:01 invader1 goes.vnetd[9061]:
Apr 3 17:59:01 invader1 goes.vnetd[9061]: goroutine 467 [running]:
Apr 3 17:59:01 invader1 goes.vnetd[9061]: panic(0xe20320, 0xc820c91270)
Apr 3 17:59:01 invader1 goes.vnetd[9061]: /usr/local/go/src/runtime/panic.go:464 +0x3e6
Apr 3 17:59:01 invader1 goes.vnetd[9061]: github.com/platinasystems/go/vnet/unix.(*netlinkMain).LoopInit(0xc820110df8, 0xc8200d2b00)
Apr 3 17:59:01 invader1 goes.vnetd[9061]: /home/stig/go/src/github.com/platinasystems/go/vnet/unix/netlink.go:176 +0x1e6
Apr 3 17:59:01 invader1 goes.vnetd[9061]: github.com/platinasystems/go/elib/loop.(*Loop).callInitNode.func1.1(0x7ff3a41f8c40, 0xc820110df8, 0xc8200d2b00, 0xc8200d2be0)
Apr 3 17:59:01 invader1 goes.vnetd[9061]: /home/stig/go/src/github.com/platinasystems/go/elib/loop/loop.go:408 +0x31
Apr 3 17:59:01 invader1 goes.vnetd[9061]: created by github.com/platinasystems/go/elib/loop.(*Loop).callInitNode.func1
Apr 3 17:59:01 invader1 goes.vnetd[9061]: /home/stig/go/src/github.com/platinasystems/go/elib/loop/loop.go:410 +0x90
Apr 3 17:59:04 invader1 goes.vnetd[9061]: exit status 2

DMAR messages on console during goes stop/start loop test

During repeated goes stop/start loop test, see following messages appear on console log occasionally. The entire log below appear on console in 1 burst.

Does not notice any obvious impact to goes.

DMAR: DRHD: handling fault status reg 2
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc0e000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc0e000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc0e000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc0e000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc02000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc02000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc02000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc02000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc02000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc03000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc03000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc03000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc03000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc03000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc03000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc03000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc03000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc03000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc03000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc03000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc03000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc03000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc03000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc03000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc03000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc03000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc03000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc03000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc03000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc03000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc03000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc03000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc03000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc04000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc04000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc04000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc04000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc04000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc04000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc04000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc04000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc04000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc04000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc04000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc04000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc04000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc04000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc04000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc04000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc04000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc04000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc04000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc04000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc04000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc04000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc04000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc04000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc05000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc05000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc05000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc05000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc05000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc05000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc05000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc05000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc05000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc05000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc05000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc05000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc05000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc05000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc05000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc05000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc05000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc05000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc05000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc05000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc05000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc05000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc05000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc05000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc05000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc06000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc06000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc06000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc06000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc06000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc06000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc06000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc06000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc06000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc06000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc06000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc06000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc06000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc06000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc06000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc06000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc06000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc06000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc06000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc06000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc06000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc06000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc06000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc06000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc07000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc07000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc07000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc07000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc07000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc07000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc07000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc07000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc07000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc07000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc07000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc07000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc07000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc07000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc07000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc07000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc07000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc07000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc07000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc07000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc07000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc07000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc07000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc07000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc08000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc08000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc08000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc08000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc08000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc08000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc08000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc08000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc08000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc08000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc08000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc08000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc08000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc08000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc08000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc08000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc08000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc08000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc08000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc08000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc08000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc08000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc08000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc08000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc09000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc09000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc09000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc09000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc09000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc09000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc09000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc09000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc09000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc09000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc09000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc09000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc09000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc09000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc09000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc09000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc09000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc09000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc09000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc09000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc09000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc09000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc09000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc09000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc0a000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc0a000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc0a000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc0a000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc0a000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc0a000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc0a000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc0a000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc0a000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc0a000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc0a000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc0a000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc0a000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc0a000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc0a000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc0a000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc0a000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc0a000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc0a000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc0a000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc0a000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc0a000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc0a000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc0a000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc0b000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc0b000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc0b000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc0b000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc0b000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc0b000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc0b000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc0b000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc0b000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc0b000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc0b000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc0b000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc0b000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc0b000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc0b000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc0b000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc0b000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc0b000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc0b000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc0b000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc0b000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc0b000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc0b000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc0b000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc0c000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc0c000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc0c000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc0c000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc0c000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc0c000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc0c000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc0c000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc0c000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc0c000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc0c000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc0c000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc0c000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc0c000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc0c000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc0c000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc0c000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc0c000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc0c000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc0c000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc0c000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc0c000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc0c000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc0c000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc0d000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc0d000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc0d000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc0d000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc0d000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc0d000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc0d000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc0d000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc0d000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc0d000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc0d000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc0d000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc0d000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc0d000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc0d000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc0d000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc0d000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc0d000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc0d000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc0d000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc0d000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc0d000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc0d000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc0d000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc0e000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc0e000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc0e000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc0e000 [fault reason 05] PTE Write access is not set
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc0e000 [fault reason 05] PTE Write access is not set
DMAR: DRHD: handling fault status reg 700

ttl=1 packets from i8 fail to arrive at i16's x86

Version:

donn@invader16:~$ /usr/bin/goes version
98f3841
(today @ 3:27pm)

Symptom: i8-i16 ping OK. But BGP between them fails.

What seems to be happening:

  • If ttl=64, packet from i8 arrives at i16.
  • If ttl=1, packet from i8 does NOT arrive at i16.

This is what i16 sees:

17:28:03.976374 IP 10.9.31.8.49567 > 10.9.31.16.179: Flags [S], seq 3153293062, win 25872, options [mss 4312,nop,nop,sackOK,nop,wscale 8], length 0
17:28:03.976492 IP 10.9.31.16.179 > 10.9.31.8.49567: Flags [S.], seq 171549443, ack 3153293063, win 29200, options [mss 1460,nop,nop,sackOK,nop,wscale 8], length 0
17:28:03.976987 IP 10.9.31.8.49567 > 10.9.31.16.179: Flags [.], ack 1, win 102, length 0
17:28:03.977485 IP 10.9.31.16.179 > 10.9.31.8.49567: Flags [F.], seq 1, ack 1, win 115, length 0
17:28:04.183474 IP 10.9.31.16.179 > 10.9.31.8.49567: Flags [F.], seq 1, ack 1, win 115, length 0
17:28:04.391692 IP 10.9.31.16.179 > 10.9.31.8.49567: Flags [F.], seq 1, ack 1, win 115, length 0
17:28:04.815426 IP 10.9.31.16.179 > 10.9.31.8.49567: Flags [F.], seq 1, ack 1, win 115, length 0
17:28:05.643531 IP 10.9.31.16.179 > 10.9.31.8.49567: Flags [F.], seq 1, ack 1, win 115, length 0
17:28:07.307491 IP 10.9.31.16.179 > 10.9.31.8.49567: Flags [F.], seq 1, ack 1, win 115, length 0
17:28:10.607528 IP 10.9.31.16.179 > 10.9.31.8.49567: Flags [F.], seq 1, ack 1, win 115, length 0
^C
16 packets captured
16 packets received by filter
0 packets dropped by kernel
donn@invader16:~$

So, i16 sees SYN and ACK packets from i8, and nothing else.

But i8 is sending other packets: Packets with ttl=1 (because this is bgp)

tcpdump on i8 during that same time:

17:28:03.921810 IP (tos 0x0, ttl 64, id 3637, offset 0, flags [DF], proto TCP (6), length 52)                                                                                              [77/17919]
    10.9.31.8.49567 > 10.9.31.16.179: Flags [S], cksum 0xbf65 (correct), seq 3153293062, win 25872, options [mss 4312,nop,nop,sackOK,nop,wscale 8], length 0
17:28:03.922628 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 52)
    10.9.31.16.179 > 10.9.31.8.49567: Flags [S.], cksum 0x103c (correct), seq 171549443, ack 3153293063, win 29200, options [mss 1460,nop,nop,sackOK,nop,wscale 8], length 0
17:28:03.922652 IP (tos 0x0, ttl 64, id 3638, offset 0, flags [DF], proto TCP (6), length 40)
    10.9.31.8.49567 > 10.9.31.16.179: Flags [.], cksum 0xc2b9 (correct), ack 1, win 102, length 0
17:28:03.922801 IP (tos 0x0, ttl 1, id 3639, offset 0, flags [DF], proto TCP (6), length 85)
    10.9.31.8.49567 > 10.9.31.16.179: Flags [P.], cksum 0x854f (correct), seq 1:46, ack 1, win 102, length 45: BGP
        Open Message (1), length: 45
          Version 4, my AS 65208, Holdtime 90s, ID 192.168.155.3
          Optional parameters, length: 16
            Option Capabilities Advertisement (2), length: 14
              Route Refresh (2), length: 0
              Multiprotocol Extensions (1), length: 4
                AFI IPv4 (1), SAFI Unicast (1)
              32-Bit AS Number (65), length: 4
                 4 Byte AS 65208
17:28:03.923468 IP (tos 0x0, ttl 64, id 33539, offset 0, flags [DF], proto TCP (6), length 40)
    10.9.31.16.179 > 10.9.31.8.49567: Flags [F.], cksum 0xc2ab (correct), seq 1, ack 1, win 115, length 0
17:28:03.923521 IP (tos 0x0, ttl 1, id 3640, offset 0, flags [DF], proto TCP (6), length 40)
    10.9.31.8.49567 > 10.9.31.16.179: Flags [F.], cksum 0xc28a (correct), seq 46, ack 2, win 102, length 0
17:28:03.932613 IP (tos 0x0, ttl 1, id 3641, offset 0, flags [DF], proto TCP (6), length 40)
    10.9.31.8.49567 > 10.9.31.16.179: Flags [F.], cksum 0xc28a (correct), seq 46, ack 2, win 102, length 0
17:28:04.129577 IP (tos 0x0, ttl 64, id 33540, offset 0, flags [DF], proto TCP (6), length 40)
    10.9.31.16.179 > 10.9.31.8.49567: Flags [F.], cksum 0xc2ab (correct), seq 1, ack 1, win 115, length 0
17:28:04.129587 IP (tos 0x0, ttl 1, id 3642, offset 0, flags [DF], proto TCP (6), length 52)
    10.9.31.8.49567 > 10.9.31.16.179: Flags [.], cksum 0x31f7 (correct), ack 2, win 102, options [nop,nop,sack 1 {1:2}], length 0
17:28:04.337782 IP (tos 0x0, ttl 64, id 33541, offset 0, flags [DF], proto TCP (6), length 40)
    10.9.31.16.179 > 10.9.31.8.49567: Flags [F.], cksum 0xc2ab (correct), seq 1, ack 1, win 115, length 0
17:28:04.337791 IP (tos 0x0, ttl 1, id 3643, offset 0, flags [DF], proto TCP (6), length 52)
    10.9.31.8.49567 > 10.9.31.16.179: Flags [.], cksum 0x31f7 (correct), ack 2, win 102, options [nop,nop,sack 1 {1:2}], length 0

Notice at 17:28:03.922801, i8 sends BGP pkt with ttl=1, which never arrives at i16.

Other packets with ttl=1 never arrive at i16.

mmu_unicast_tx_cos0_drop_packets on good ports

Setup:

pkgen <-> (ports 2,4,6...32) switch2 (ports 1,3,5..31) <-> (ports 1,3,5...31) switch17 (ports 2,4,6...32) <-> pkgen

Traffic flowing bidirectionally using static ARP on the switches. No drops.
Ports in autoneg mode.

Physically disconnect cable on port 27 between the 2 switches.
Link goes down on port 27 on both switches, expected
vnet.eth-27-1. mmu_unicast_tx_cos0_drop_packets increments on both switches, expected.
mmu_unicast_tx_cos0_drop_packets increments also on ports eth-16 through eth-32 of both switches, unexpected

With cable still disconnected, set eth-27-1.speed to fixed speed (100g) on switch17
mmu_unicast_tx_cos0_drop_packets stops incrementing on all ports eth-16 through eth-32, including eth-27 on switch17

With cable still disconnected, set eth-27-1.speed to autoneg on switch17
mmu_unicast_tx_cos0_drop_packets starts incrementing on all ports eth-16 through eth-32 again on switch17 again

Why is behavior different between autoneg and fixed speed? Seems like behavior with fixed speed is the correct behavior.

vnet crashes when adding 7th path to ECMP

Version: 74dfc8b
(also included commit c7d5054515784cb33860ef15aa5e5f4d17fdd4ea in fe1)

Always reproducible.

10.0.0.2, 10.0.2.2, 10.0.4.2, ... ,10.0.30.2 have static arp set up.
e.g. 10.0.0.2 dev eth-0-0 lladdr 02:46:8a:00:03:ae PERMANENT

Steps to reproduce:
fyang@invader7:$ sudo ip route add 100.0.0.0/24 via 10.0.0.2
fyang@invader7:$ sudo ip route app 100.0.0.0/24 via 10.0.2.2
fyang@invader7:$ sudo ip route app 100.0.0.0/24 via 10.0.4.2
fyang@invader7:$ sudo ip route app 100.0.0.0/24 via 10.0.6.2
fyang@invader7:$ sudo ip route app 100.0.0.0/24 via 10.0.8.2
fyang@invader7:$ sudo ip route app 100.0.0.0/24 via 10.0.10.2
fyang@invader7:$ sudo ip route app 100.0.0.0/24 via 10.0.12.2 <---- vnet crashed

Restart goes and retry with different ECMP paths
fyang@invader7:$ sudo ip route add 100.0.0.0/24 via 10.0.8.2
fyang@invader7:$ sudo ip route app 100.0.0.0/24 via 10.0.10.2
fyang@invader7:$ sudo ip route app 100.0.0.0/24 via 10.0.12.2
fyang@invader7:$ sudo ip route app 100.0.0.0/24 via 10.0.14.2
fyang@invader7:$ sudo ip route app 100.0.0.0/24 via 10.0.16.2
fyang@invader7:$ sudo ip route app 100.0.0.0/24 via 10.0.18.2
fyang@invader7:$ sudo ip route app 100.0.0.0/24 via 10.0.20.2 <---- vnet crashed

Syslog:
Apr 28 12:24:29 invader7 goes.vnetd[31046]: unknown multipath adjacency: goroutine 52 [running]:
Apr 28 12:24:29 invader7 goes.vnetd[31046]: runtime/debug.Stack(0xc420869a40, 0xa54b20, 0xc420ab4990)
Apr 28 12:24:29 invader7 goes.vnetd[31046]: /usr/local/go/src/runtime/debug/stack.go:24 +0x79
Apr 28 12:24:29 invader7 goes.vnetd[31046]: github.com/platinasystems/go/elib/loop.(*Loop).doEvent.func1(0xc42018a000)
Apr 28 12:24:29 invader7 goes.vnetd[31046]: /home/fyang/gopath/src/github.com/platinasystems/go/elib/loop/event.go:128 +0x91
Apr 28 12:24:29 invader7 goes.vnetd[31046]: panic(0xa54b20, 0xc420ab4990)
Apr 28 12:24:29 invader7 goes.vnetd[31046]: /usr/local/go/src/runtime/panic.go:458 +0x243
Apr 28 12:24:29 invader7 goes.vnetd[31046]: github.com/platinasystems/go/vnet/ip.(*multipathAdjacency).free(0xc420ca6530, 0xc42021e068)
Apr 28 12:24:29 invader7 goes.vnetd[31046]: /home/fyang/gopath/src/github.com/platinasystems/go/vnet/ip/adjacency.go:662 +0x290
Apr 28 12:24:29 invader7 goes.vnetd[31046]: github.com/platinasystems/go/vnet/ip.(*Main).AddDelNextHop(0xc42021e068, 0x25a, 0x1000000b3, 0xc42009
8b28)
Apr 28 12:24:29 invader7 goes.vnetd[31046]: /home/fyang/gopath/src/github.com/platinasystems/go/vnet/ip/adjacency.go:537 +0x2ad
Apr 28 12:24:29 invader7 goes.vnetd[31046]: github.com/platinasystems/go/vnet/ip4.(*Main).AddDelRouteNextHop(0xc42021e000, 0xc420c23bf0, 0xc420c2
3c00, 0x34020c0000, 0x1, 0xc4201880f0)
Apr 28 12:24:29 invader7 goes.vnetd[31046]: /home/fyang/gopath/src/github.com/platinasystems/go/vnet/ip4/fib.go:429 +0x1d8
Apr 28 12:24:29 invader7 goes.vnetd[31046]: github.com/platinasystems/go/vnet/unix.(*Main).ip4RouteMsg(0xc420224500, 0xc420bb11e0, 0xc42007e401,
0xc4203f5d40, 0xc420869e70)
Apr 28 12:24:29 invader7 goes.vnetd[31046]: /home/fyang/gopath/src/github.com/platinasystems/go/vnet/unix/netlink.go:542 +0x29e
Apr 28 12:24:29 invader7 goes.vnetd[31046]: github.com/platinasystems/go/vnet/unix.(*netlinkEvent).EventAction(0xc420b0c840)
Apr 28 12:24:29 invader7 goes.vnetd[31046]: /home/fyang/gopath/src/github.com/platinasystems/go/vnet/unix/netlink.go:345 +0x47e
Apr 28 12:24:29 invader7 goes.vnetd[31046]: github.com/platinasystems/go/elib/loop.(*loopEvent).do(0xc4204d6cc0)
Apr 28 12:24:29 invader7 goes.vnetd[31046]: /home/fyang/gopath/src/github.com/platinasystems/go/elib/loop/event.go:118 +0x34
Apr 28 12:24:29 invader7 goes.vnetd[31046]: github.com/platinasystems/go/elib/loop.(*Loop).doEvent(0xc42018a000, 0xc4204d6cc0)
Apr 28 12:24:29 invader7 goes.vnetd[31046]: /home/fyang/gopath/src/github.com/platinasystems/go/elib/loop/event.go:133 +0x51
Apr 28 12:24:29 invader7 goes.vnetd[31046]: github.com/platinasystems/go/elib/loop.(*Loop).eventHandler(0xc42018a000, 0x12d7280, 0xc42018a548)
Apr 28 12:24:29 invader7 goes.vnetd[31046]: /home/fyang/gopath/src/github.com/platinasystems/go/elib/loop/event.go:140 +0x86
Apr 28 12:24:29 invader7 goes.vnetd[31046]: created by github.com/platinasystems/go/elib/loop.(*Loop).startHandler
Apr 28 12:24:29 invader7 goes.vnetd[31046]: /home/fyang/gopath/src/github.com/platinasystems/go/elib/loop/event.go:153 +0x124
Apr 28 12:24:29 invader7 goes.vnetd[31046]: done

deleting dummy interface causes panic

Code up-to-date with today's fix which seem to fix #23 (and maybe #22 and #25), however when I delete the dummy interface it panics (sometimes "ip link del dummy0" also causes it.

root@invader1:~# goes vnet show ip fib
Table Destination Adjacency

root@invader1:# ip link set up dev eth-1-0
root@invader1:# ip add add 1.1.1.1/23 dev eth-1-0

root@invader1:~# goes vnet show ip fib
Table Destination Adjacency
0 1.1.0.0/23 3: glean eth-1-0
0 1.1.1.1/32 4: local eth-1-0

root@invader1:# ip link add name dummy0 type dummy
root@invader1:# ip link set up dummy0
root@invader1:~# ip add add 3.3.3.3/24 dev dummy0

root@invader1:~# goes vnet show ip fib
Table Destination Adjacency
0 1.1.0.0/23 3: glean eth-1-0
0 1.1.1.1/32 4: local eth-1-0
0 3.3.3.0/24 2: punt

root@invader1:~# ip link del dummy0

  • causes

Mar 8 14:46:04 invader1 goes.vnetd[19385]: not found: goroutine 36 [running]:
Mar 8 14:46:04 invader1 goes.vnetd[19385]: runtime/debug.Stack(0x0, 0x0, 0x0)
Mar 8 14:46:04 invader1 goes.vnetd[19385]: /usr/local/go/src/runtime/debug/stack.go:24 +0x80
Mar 8 14:46:04 invader1 goes.vnetd[19385]: github.com/platinasystems/go/elib/loop.(*Loop).doEvent.func1(0xc8200f4b00)
Mar 8 14:46:04 invader1 goes.vnetd[19385]: /home/stig/go/src/github.com/platinasystems/go/elib/loop/event.go:128 +0xa2
Mar 8 14:46:04 invader1 goes.vnetd[19385]: panic(0xb94ea0, 0xc82000b170)
Mar 8 14:46:04 invader1 goes.vnetd[19385]: /usr/local/go/src/runtime/panic.go:426 +0x4e9
Mar 8 14:46:04 invader1 goes.vnetd[19385]: github.com/platinasystems/go/vnet/devices/ethernet/switch/fe1/internal/fe1a.(*fe1a).ip4_fib_add_del(0xc8201f8000, 0xc800000000, 0xc82000b118, 0x100000002)
Mar 8 14:46:04 invader1 goes.vnetd[19385]: /home/stig/go/src/github.com/platinasystems/go/vnet/devices/ethernet/switch/fe1/internal/fe1a/fib.go:279 +0x1a0
Mar 8 14:46:04 invader1 goes.vnetd[19385]: github.com/platinasystems/go/vnet/devices/ethernet/switch/fe1/internal/fe1a.(*fe1a).(github.com/platinasystems/go/vnet/devices/ethernet/switch/fe1/internal/fe1a.ip4_fib_add_del)-fm(0xc800000000, 0xc82000b118, 0x100000002)
Mar 8 14:46:04 invader1 goes.vnetd[19385]: /home/stig/go/src/github.com/platinasystems/go/vnet/devices/ethernet/switch/fe1/internal/fe1a/fib.go:49 +0x4c
Mar 8 14:46:04 invader1 goes.vnetd[19385]: github.com/platinasystems/go/vnet/ip4.(*Fib).addDel(0xc820486160, 0xc8201d0000, 0xc82000b118, 0x100000002, 0xb4ac80)
Mar 8 14:46:04 invader1 goes.vnetd[19385]: /home/stig/go/src/github.com/platinasystems/go/vnet/ip4/fib.go:176 +0xb9
Mar 8 14:46:04 invader1 goes.vnetd[19385]: github.com/platinasystems/go/vnet/ip4.(*Main).addDelRoute(0xc8201d0000, 0xc820cdf3c0, 0x200000000, 0x1, 0xdcfc80, 0x0, 0x0)
Mar 8 14:46:04 invader1 goes.vnetd[19385]: /home/stig/go/src/github.com/platinasystems/go/vnet/ip4/fib.go:367 +0x1cf
Mar 8 14:46:04 invader1 goes.vnetd[19385]: github.com/platinasystems/go/vnet/ip4.(*Main).(github.com/platinasystems/go/vnet/ip4.addDelRoute)-fm(0xc820cdf3c0, 0x200000000, 0x1, 0xc81ff82e8e, 0x0, 0x0)
Mar 8 14:46:04 invader1 goes.vnetd[19385]: /home/stig/go/src/github.com/platinasystems/go/vnet/ip4/package.go:24 +0x51
Mar 8 14:46:04 invader1 goes.vnetd[19385]: github.com/platinasystems/go/vnet/unix.(*dummyInterface).addDelDummyPuntPrefixes(0xc820b06060, 0xc8201a8480, 0xc800002e01)
Mar 8 14:46:04 invader1 goes.vnetd[19385]: /home/stig/go/src/github.com/platinasystems/go/vnet/unix/netlink.go:93 +0x30b
Mar 8 14:46:04 invader1 goes.vnetd[19385]: github.com/platinasystems/go/vnet/unix.(*netlinkEvent).EventAction(0xc8208a8060)
Mar 8 14:46:04 invader1 goes.vnetd[19385]: /home/stig/go/src/github.com/platinasystems/go/vnet/unix/netlink.go:283 +0x813
Mar 8 14:46:04 invader1 goes.vnetd[19385]: github.com/platinasystems/go/elib/loop.(*loopEvent).do(0xc8208a8390)
Mar 8 14:46:04 invader1 goes.vnetd[19385]: /home/stig/go/src/github.com/platinasystems/go/elib/loop/event.go:118 +0x135
Mar 8 14:46:04 invader1 goes.vnetd[19385]: github.com/platinasystems/go/elib/loop.(*Loop).doEvent(0xc8200f4b00, 0xc8208a8390)
Mar 8 14:46:04 invader1 goes.vnetd[19385]: /home/stig/go/src/github.com/platinasystems/go/elib/loop/event.go:133 +0x48
Mar 8 14:46:04 invader1 goes.vnetd[19385]: github.com/platinasystems/go/elib/loop.(*Loop).eventHandler(0xc8200f4b00, 0x7f038a8753a0, 0xc8200f5048)
Mar 8 14:46:04 invader1 goes.vnetd[19385]: /home/stig/go/src/github.com/platinasystems/go/elib/loop/event.go:140 +0x81
Mar 8 14:46:04 invader1 goes.vnetd[19385]: created by github.com/platinasystems/go/elib/loop.(*Loop).startHandler
Mar 8 14:46:04 invader1 goes.vnetd[19385]: /home/stig/go/src/github.com/platinasystems/go/elib/loop/event.go:153 +0x120

Panic after 'sh ip fib'

Build version:

commit 71edbc1d0d43cdeb99cd618db30d0cb33d8ef355
Author: Tom Grennan <[email protected]>
Date:   Fri Dec 30 17:34:31 2016 -0800

(pulled and built on 1/3/2017)

After running a local script that adds (and subsequently deletes) 1536 static routes w/ "ip route add ..." / "ip route del ...", after ~15mins my script detected FIB inconsistency (rib=0, fib=1 route). Returning after lunch and seeing this, I did:

donn@invader8:~$ sudo goes vnet sh ip fib
[sudo] password for donn:
 Table                   Destination           Adjacency
     0              192.168.155.0/24  1: glean eth-31-0
     0              192.168.155.2/32  3: rewrite eth-31-0 IP4: 02:46:8a:00:05:b4 -> 00:05:64:2f:3c:c9
     0              192.168.155.3/32  2: local eth-31-0
                                        tx pipe packets                2
                                        tx pipe bytes                196
donn@invader8:~$

And vnet must have died, because a few secs later I did 'sh ip fib' again and saw this:

donn@invader8:~$ sudo goes vnet sh ip fib | egrep " 20\.[0-9]+" | wc -l
vnet: dial unix /run/goes/socks/vnet: connect: connection refused
0
donn@invader8:~$

Then saw this in syslog:

Jan  5 12:26:34 invader8 goes.vnetd[31085]: panic: write broken pipe
Jan  5 12:26:34 invader8 goes.vnetd[31085]:
Jan  5 12:26:34 invader8 goes.vnetd[31085]: goroutine 12 [running]:
Jan  5 12:26:34 invader8 goes.vnetd[31085]: panic(0xa18aa0, 0xc4225fc0a0)
Jan  5 12:26:34 invader8 goes.vnetd[31085]:         /usr/local/go/src/runtime/panic.go:500 +0x1a1
Jan  5 12:26:34 invader8 goes.vnetd[31085]: github.com/platinasystems/go/elib/iomux.(*Mux).do(0x12253a0, 0xc420045350)
Jan  5 12:26:34 invader8 goes.vnetd[31085]:         /home/donn/workspace/go/src/github.com/platinasystems/go/elib/iomux/epoll.go:143 +0x186
Jan  5 12:26:34 invader8 goes.vnetd[31085]: github.com/platinasystems/go/elib/iomux.(*Mux).EventPoll(0x12253a0)
Jan  5 12:26:34 invader8 goes.vnetd[31085]:         /home/donn/workspace/go/src/github.com/platinasystems/go/elib/iomux/epoll.go:166 +0xde
Jan  5 12:26:34 invader8 goes.vnetd[31085]: github.com/platinasystems/go/elib/loop.(*eventLoop).eventPoller(0xc4200cee68, 0x12054e0, 0x12253a0)
Jan  5 12:26:34 invader8 goes.vnetd[31085]:         /home/donn/workspace/go/src/github.com/platinasystems/go/elib/loop/event.go:156 +0x31
Jan  5 12:26:34 invader8 goes.vnetd[31085]: created by github.com/platinasystems/go/elib/loop.(*eventLoop).startPoller
Jan  5 12:26:34 invader8 goes.vnetd[31085]:         /home/donn/workspace/go/src/github.com/platinasystems/go/elib/loop/event.go:159 +0x53
Jan  5 12:26:36 invader8 goes.vnetd[31085]: exit status 2

which agrees with the date/time of my 1st 'sh ip fib'.

Complete syslog is stored in invader8:~donn/syslog_panic_2017_01_05.txt

Steps to reproduce:
Use a script to add 1536 /31 routes to kernel RIB.
Have script delete those routes.
Pause 5 seconds.
Compare RIB and FIB.
If FIB is in sync with RIB, repeat test.

Another observation: I suspect that vnet was lagging and not keeping sync with kernel's route table because my script pauses 5 seconds before comparing the kernel RIB to vnet FIB.
After 5 secs, the number of RIB routes was 0 (correct), but the FIB route count was 1 (incorrect, should be zero after all 1536 routes were deleted).
When I issued 'sh ip fib' manually (an hr later and before vnet died), it showed 0 of the test routes (20.x.x.x/31 routes), which is correct. But then vnet died.

/run/goes/socks/vnet group and permissions

/run/goes/socks/vnet should have group vnet and 0775 permissions like the other sockets.

$ ls -l /run/goes/socks
total 0
srwxrwxr-x 1 root adm 0 Jan 26 15:10 nld
srwxrwxr-x 1 root adm 0 Jan 26 15:10 redisd
srwxrwxr-x 1 root adm 0 Jan 26 15:10 redis-pub
srwxrwxr-x 1 root adm 0 Jan 26 15:10 redis-reg
srwxr-xr-x 1 root root 0 Jan 26 15:10 vnet
srwxrwxr-x 1 root adm 0 Jan 26 15:10 vnetd

"pool: too large" error

What does this error mean?

Had about 1500 routes in fib
Did 'vnet show ip fib'
Did ifdown, ifup
Did some autoneg cmds
Noticed vnet process died
Saw this in syslog:

Jan 24 08:05:22 invader8 goes.vnetd[14587]: pool: too large
Jan 24 08:05:22 invader8 goes.vnetd[14587]: not found
Jan 24 08:05:22 invader8 goes.vnetd[14587]: pool: too large
Jan 24 08:05:24 invader8 goes.vnetd[14587]: done

donn@invader8:~$ ps auxw | grep "vnet"
donn     14390  0.0  0.0  12732  2164 pts/0    S+   09:36   0:00 grep vnet
donn@invader8:~$ goes start
start: already started

These were the ifdown/up and autoneg cmds:

                "+ /sbin/ifdown eth-31-0",
                "+ /sbin/ifup eth-31-0",
                "+ /usr/bin/goes vnet set ha sp eth-31-0 auto",
                "+ /usr/bin/goes vnet set ha sp eth-31-0 100g",
                "+ /usr/bin/goes vnet set ha sp eth-31-0 auto dis",
                "+ /usr/bin/goes vnet set ha sp eth-31-0 auto ena"

Note: The goes I'm running is the patch:

go/internal/optional/vnet/vnet.go: line 43

  •            var buf [4096]byte

  •           var buf [4*4096]byte

Eviction fifo panic

"Pipe counter wrap" panic. Need to reproduce in controlled test-case and figure out why the fifo is wrapping at all.

fatal error: concurrent map read and map write

Caught this once during “goes start”; very rare. Had an overnight loop testing something else with 2500 loops of stop/start on Invader7, and saw this just once.

This is happening in redisd.go around.

func (cmd *cmd) Main(args ...string)
…
err = Hook(pub)
pub <- "redis.ready: true"
close(pub)
…

More often I’ve seen this fail with just returning err = ““read: no such device or address” on invader2.
Logging as a separate issue for now, though could be related.

Syslog (extra debug prints):

Jan 5 22:04:29 invader7 goes.redisd[17309]: goes.Main []
Jan 5 22:04:29 invader7 goes.redisd[17309]: new arg = [redisd]
Jan 5 22:04:29 invader7 goes.redisd[17309]: next command g := byName[redisd]
Jan 5 22:04:29 invader7 goes.redisd[17309]: g.Kind.IsDaemon() = true
Jan 5 22:04:29 invader7 goes.redisd[17309]: redisd.main Hello==========================
Jan 5 22:04:29 invader7 goes.redisd[17309]: redisd.main finished varrun stuffs==========================
Jan 5 22:04:29 invader7 goes.redisd[17309]: redisd.main finished starting server===/run/goes/socks/redisd
Jan 5 22:04:29 invader7 goes.redisd[17309]: redisd.main finished chgroup adm===/run/goes/socks/redisd
Jan 5 22:04:29 invader7 goes.redisd[17309]: redisd.main finished assign, unassign===/run/goes/socks/redisd
Jan 5 22:04:29 invader7 goes.redisd[17309]: redisd.main published hostname = invader7
Jan 5 22:04:29 invader7 goes.redisd[17309]: redisd.main published machine = platina-mk1
Jan 5 22:04:29 invader7 goes.redisd[17309]: redisd.main published cmdline.BOOT_IMAGE = /boot/vmlinuz-4.6.0-rc2-platina-mk1-amd64
Jan 5 22:04:29 invader7 goes.redisd[17309]: redisd.main published cmdline.console = ttyS0,115200
Jan 5 22:04:29 invader7 goes.redisd[17309]: listen: /run/goes/socks/redisd
Jan 5 22:04:29 invader7 goes.redisd[17309]: redisd.main published cmdline.quiet = true
Jan 5 22:04:29 invader7 goes.redisd[17309]: redisd.main published cmdline.ro = true
Jan 5 22:04:29 invader7 goes.redisd[17309]: redisd.main published cmdline.root = UUID=93576419-075c-46f1-a4b6-8b71ba5de647
Jan 5 22:04:29 invader7 goes.redisd[17309]: redisd.main published cmdline.video = off
Jan 5 22:04:29 invader7 goes.redisd[17309]: listen: [127.0.0.1%lo]:6379
Jan 5 22:04:29 invader7 goes.redisd[17309]: listen: [::1%lo]:6379
Jan 5 22:04:29 invader7 goes.redisd[17309]: listen: [192.168.101.127%eth0]:6379
Jan 5 22:04:29 invader7 goes.redisd[17309]: listen: [fe80::46:8aff:fe00:9f4%eth0]:6379
Jan 5 22:04:30 invader7 goes.redisd[17309]: fatal error: concurrent map read and map write
Jan 5 22:04:30 invader7 goes.redisd[17309]:
Jan 5 22:04:30 invader7 goes.redisd[17309]: goroutine 7 [running]:
Jan 5 22:04:30 invader7 goes.redisd[17309]: runtime.throw(0xb332a7, 0x21)
Jan 5 22:04:30 invader7 goes.redisd[17309]: /usr/local/go/src/runtime/panic.go:566 +0x95 fp=0xc420047660 sp=0xc420047640
Jan 5 22:04:30 invader7 goes.redisd[17309]: runtime.mapaccess1_faststr(0xa39740, 0xc4201a2a50, 0xc42024ce80, 0x15, 0x1)
Jan 5 22:04:30 invader7 goes.redisd[17309]: /usr/local/go/src/runtime/hashmap_fast.go:201 +0x4f3 fp=0xc4200476c0 sp=0xc420047660
Jan 5 22:04:30 invader7 goes.redisd[17309]: github.com/platinasystems/go/internal/required/redisd.(*Redisd).Hget(0xc420166120, 0xc420239bc0, 0x7, 0xc42024ce80, 0x15, 0x0, 0x0, 0x0, 0x0, 0x0)
Jan 5 22:04:30 invader7 goes.redisd[17309]: /home/fyang/gopath/src/github.com/platinasystems/go/internal/required/redisd/redisd.go:461 +0x124 fp=0xc4200477e8 sp=0xc4200476c0
Jan 5 22:04:30 invader7 goes.redisd[17309]: runtime.call128(0xc42006fce0, 0xc420136320, 0xc420055220, 0x2800000050)
Jan 5 22:04:30 invader7 goes.redisd[17309]: /usr/local/go/src/runtime/asm_amd64.s:481 +0x66 fp=0xc420047878 sp=0xc4200477e8
Jan 5 22:04:30 invader7 goes.redisd[17309]: reflect.Value.call(0xc42014a380, 0xc420136320, 0x13, 0xb1b482, 0x4, 0xc42006b320, 0x3, 0x4, 0x9dc4c0, 0xc420239c20, ...)
Jan 5 22:04:30 invader7 goes.redisd[17309]: /usr/local/go/src/reflect/value.go:434 +0x5c8 fp=0xc420047bc8 sp=0xc420047878
Jan 5 22:04:30 invader7 goes.redisd[17309]: reflect.Value.Call(0xc42014a380, 0xc420136320, 0x13, 0xc42006b320, 0x3, 0x4, 0x54, 0x1f, 0x1f)
Jan 5 22:04:30 invader7 goes.redisd[17309]: /usr/local/go/src/reflect/value.go:302 +0xa4 fp=0xc420047c30 sp=0xc420047bc8
Jan 5 22:04:30 invader7 goes.redisd[17309]: github.com/platinasystems/go/vendor/github.com/platinasystems/go-redis-server.(*Server).handlerFn.func1(0xc420236600, 0xc420133e30, 0xc420239bbc, 0x4, 0xc4201dacb8)
Jan 5 22:04:30 invader7 goes.redisd[17309]: /home/fyang/gopath/src/github.com/platinasystems/go/vendor/github.com/platinasystems/go-redis-server/auto.go:94 +0xb02 fp=0xc420047e48 sp=0xc420047c30
Jan 5 22:04:30 invader7 goes.redisd[17309]: github.com/platinasystems/go/vendor/github.com/platinasystems/go-redis-server.(*Server).Apply(0xc420202780, 0xc420236600, 0x0, 0x0, 0x0, 0x0)
Jan 5 22:04:30 invader7 goes.redisd[17309]: /home/fyang/gopath/src/github.com/platinasystems/go/vendor/github.com/platinasystems/go-redis-server/handler.go:42 +0xfc fp=0xc420047e88 sp=0xc420047e48
Jan 5 22:04:30 invader7 goes.redisd[17309]: github.com/platinasystems/go/vendor/github.com/platinasystems/go-redis-server.(*Server).ServeClient(0xc420202780, 0x1212560, 0xc4200241c0, 0x0, 0x0)
Jan 5 22:04:30 invader7 goes.redisd[17309]: /home/fyang/gopath/src/github.com/platinasystems/go/vendor/github.com/platinasystems/go-redis-server/server.go:133 +0x24e fp=0xc420047f78 sp=0xc420047e88
Jan 5 22:04:30 invader7 goes.redisd[17309]: runtime.goexit()
Jan 5 22:04:30 invader7 goes.redisd[17309]: /usr/local/go/src/runtime/asm_amd64.s:2086 +0x1 fp=0xc420047f80 sp=0xc420047f78
Jan 5 22:04:30 invader7 goes.redisd[17309]: created by github.com/platinasystems/go/vendor/github.com/platinasystems/go-redis-server.(*Server).Serve
Jan 5 22:04:30 invader7 goes.redisd[17309]: /home/fyang/gopath/src/github.com/platinasystems/go/vendor/github.com/platinasystems/go-redis-server/server.go:82 +0xf8
Jan 5 22:04:30 invader7 goes.redisd[17309]:
Jan 5 22:04:30 invader7 goes.redisd[17309]: goroutine 1 [chan send]:
Jan 5 22:04:30 invader7 goes.redisd[17309]: main.getEepromData(0xc4201a6480, 0x26, 0xc4200459c8)
Jan 5 22:04:30 invader7 goes.redisd[17309]: /home/fyang/gopath/src/github.com/platinasystems/go/main/goes-platina-mk1/main.go:164 +0x748
Jan 5 22:04:30 invader7 goes.redisd[17309]: github.com/platinasystems/go/internal/required/redisd.(*cmd).Main(0xc420166120, 0xc42000c220, 0x2, 0x2, 0x0, 0x0)
Jan 5 22:04:30 invader7 goes.redisd[17309]: /home/fyang/gopath/src/github.com/platinasystems/go/internal/required/redisd/redisd.go:242 +0xf97
Jan 5 22:04:30 invader7 goes.redisd[17309]: github.com/platinasystems/go/internal/goes.(mainer).Main-fm(0xc42000c220, 0x0, 0x0, 0xc420167e60, 0xc420045b70)
Jan 5 22:04:30 invader7 goes.redisd[17309]: /home/fyang/gopath/src/github.com/platinasystems/go/internal/goes/goes.go:245 +0x4d
Jan 5 22:04:30 invader7 goes.redisd[17309]: github.com/platinasystems/go/internal/goes.ByName.Main(0xc42013e720, 0xc42000c220, 0x0, 0x0, 0x0, 0x0)
Jan 5 22:04:30 invader7 goes.redisd[17309]: /home/fyang/gopath/src/github.com/platinasystems/go/internal/goes/goes.go:214 +0xbb0
Jan 5 22:04:30 invader7 goes.redisd[17309]: main.main()
Jan 5 22:04:30 invader7 goes.redisd[17309]: /home/fyang/gopath/src/github.com/platinasystems/go/main/goes-platina-mk1/main.go:85 +0x442
Jan 5 22:04:30 invader7 goes.redisd[17309]:
Jan 5 22:04:30 invader7 goes.redisd[17309]: goroutine 17 [syscall, locked to thread]:
Jan 5 22:04:30 invader7 goes.redisd[17309]: runtime.goexit()
Jan 5 22:04:30 invader7 goes.redisd[17309]: /usr/local/go/src/runtime/asm_amd64.s:2086 +0x1
Jan 5 22:04:30 invader7 goes.redisd[17309]:
Jan 5 22:04:30 invader7 goes.redisd[17309]: goroutine 5 [syscall]:
Jan 5 22:04:30 invader7 goes.redisd[17309]: os/signal.signal_recv(0x0)
Jan 5 22:04:30 invader7 goes.redisd[17309]: /usr/local/go/src/runtime/sigqueue.go:116 +0x157
Jan 5 22:04:30 invader7 goes.redisd[17309]: os/signal.loop()
Jan 5 22:04:30 invader7 goes.redisd[17309]: /usr/local/go/src/os/signal/signal_unix.go:22 +0x22
Jan 5 22:04:30 invader7 goes.redisd[17309]: created by os/signal.init.1
Jan 5 22:04:30 invader7 goes.redisd[17309]: /usr/local/go/src/os/signal/signal_unix.go:28 +0x41
Jan 5 22:04:30 invader7 goes.redisd[17309]:
Jan 5 22:04:30 invader7 goes.redisd[17309]: goroutine 18 [select, locked to thread]:
Jan 5 22:04:30 invader7 goes.redisd[17309]: runtime.gopark(0xb8ee70, 0x0, 0xb1dfbe, 0x6, 0x18, 0x2)
Jan 5 22:04:30 invader7 goes.redisd[17309]: /usr/local/go/src/runtime/proc.go:259 +0x13a
Jan 5 22:04:30 invader7 goes.redisd[17309]: runtime.selectgoImpl(0xc420022730, 0x0, 0x18)
Jan 5 22:04:30 invader7 goes.redisd[17309]: /usr/local/go/src/runtime/select.go:423 +0x11d9
Jan 5 22:04:30 invader7 goes.redisd[17309]: runtime.selectgo(0xc420022730)
Jan 5 22:04:30 invader7 goes.redisd[17309]: /usr/local/go/src/runtime/select.go:238 +0x1c
Jan 5 22:04:30 invader7 goes.redisd[17309]: runtime.ensureSigM.func1()
Jan 5 22:04:30 invader7 goes.redisd[17309]: /usr/local/go/src/runtime/signal1_unix.go:304 +0x2f3
Jan 5 22:04:30 invader7 goes.redisd[17309]: runtime.goexit()
Jan 5 22:04:30 invader7 goes.redisd[17309]: /usr/local/go/src/runtime/asm_amd64.s:2086 +0x1
Jan 5 22:04:30 invader7 goes.redisd[17309]:
Jan 5 22:04:30 invader7 goes.redisd[17309]: goroutine 19 [chan receive]:
Jan 5 22:04:30 invader7 goes.redisd[17309]: github.com/platinasystems/go/internal/goes.(*Goes).wait(0xc420167440, 0xc420167e60)
Jan 5 22:04:30 invader7 goes.redisd[17309]: /home/fyang/gopath/src/github.com/platinasystems/go/internal/goes/goes.go:279 +0x70
Jan 5 22:04:30 invader7 goes.redisd[17309]: created by github.com/platinasystems/go/internal/goes.ByName.Main
Jan 5 22:04:30 invader7 goes.redisd[17309]: /home/fyang/gopath/src/github.com/platinasystems/go/internal/goes/goes.go:212 +0xe9c
Jan 5 22:04:30 invader7 goes.redisd[17309]:
Jan 5 22:04:30 invader7 goes.redisd[17309]: goroutine 20 [IO wait]:
Jan 5 22:04:30 invader7 goes.redisd[17309]: net.runtime_pollWait(0x7f9a30bfa658, 0x72, 0x0)
Jan 5 22:04:30 invader7 goes.redisd[17309]: /usr/local/go/src/runtime/netpoll.go:160 +0x59
Jan 5 22:04:30 invader7 goes.redisd[17309]: net.(*pollDesc).wait(0xc420168450, 0x72, 0xc42001ce40, 0xc42000c160)
Jan 5 22:04:30 invader7 goes.redisd[17309]: /usr/local/go/src/net/fd_poll_runtime.go:73 +0x38
Jan 5 22:04:30 invader7 goes.redisd[17309]: net.(*pollDesc).waitRead(0xc420168450, 0x120a560, 0xc42000c160)
Jan 5 22:04:30 invader7 goes.redisd[17309]: /usr/local/go/src/net/fd_poll_runtime.go:78 +0x34
Jan 5 22:04:30 invader7 goes.redisd[17309]: net.(*netFD).accept(0xc4201683f0, 0x0, 0x1208c60, 0xc4200d3200)
Jan 5 22:04:30 invader7 goes.redisd[17309]: /usr/local/go/src/net/fd_unix.go:419 +0x238
Jan 5 22:04:30 invader7 goes.redisd[17309]: net.(*UnixListener).accept(0xc420147b60, 0x0, 0x0, 0x0)
Jan 5 22:04:30 invader7 goes.redisd[17309]: /usr/local/go/src/net/unixsock_posix.go:158 +0x32
Jan 5 22:04:30 invader7 goes.redisd[17309]: net.(*UnixListener).Accept(0xc420147b60, 0xc420147b20, 0x19, 0x0, 0x0)
Jan 5 22:04:30 invader7 goes.redisd[17309]: /usr/local/go/src/net/unixsock.go:229 +0x49
Jan 5 22:04:30 invader7 goes.redisd[17309]: github.com/platinasystems/go/internal/sockfile.(*RpcServer).listen(0xc4201b4990)
Jan 5 22:04:30 invader7 goes.redisd[17309]: /home/fyang/gopath/src/github.com/platinasystems/go/internal/sockfile/sockfile.go:132 +0x7b
Jan 5 22:04:30 invader7 goes.redisd[17309]: created by github.com/platinasystems/go/internal/sockfile.NewRpcServer
Jan 5 22:04:30 invader7 goes.redisd[17309]: /home/fyang/gopath/src/github.com/platinasystems/go/internal/sockfile/sockfile.go:96 +0xe3
Jan 5 22:04:30 invader7 goes.redisd[17309]:
Jan 5 22:04:30 invader7 goes.redisd[17309]: goroutine 22 [runnable]:
Jan 5 22:04:30 invader7 goes.redisd[17309]: github.com/platinasystems/go/internal/required/redisd.(*cmd).Main.func2(0xc420166120, 0xc4201a6480)
Jan 5 22:04:30 invader7 goes.redisd[17309]: /home/fyang/gopath/src/github.com/platinasystems/go/internal/required/redisd/redisd.go:210 +0x5d
Jan 5 22:04:30 invader7 goes.redisd[17309]: created by github.com/platinasystems/go/internal/required/redisd.(*cmd).Main
Jan 5 22:04:30 invader7 goes.redisd[17309]: /home/fyang/gopath/src/github.com/platinasystems/go/internal/required/redisd/redisd.go:215 +0xa9f
Jan 5 22:04:30 invader7 goes.redisd[17309]:
Jan 5 22:04:30 invader7 goes.redisd[17309]: goroutine 34 [IO wait]:
Jan 5 22:04:30 invader7 goes.redisd[17309]: net.runtime_pollWait(0x7f9a30bfa598, 0x72, 0x0)
Jan 5 22:04:30 invader7 goes.redisd[17309]: /usr/local/go/src/runtime/netpoll.go:160 +0x59
Jan 5 22:04:30 invader7 goes.redisd[17309]: net.(*pollDesc).wait(0xc420204060, 0x72, 0xc42001d640, 0xc42000c160)
Jan 5 22:04:30 invader7 goes.redisd[17309]: /usr/local/go/src/net/fd_poll_runtime.go:73 +0x38
Jan 5 22:04:30 invader7 goes.redisd[17309]: net.(*pollDesc).waitRead(0xc420204060, 0x120a560, 0xc42000c160)
Jan 5 22:04:30 invader7 goes.redisd[17309]: /usr/local/go/src/net/fd_poll_runtime.go:78 +0x34
Jan 5 22:04:30 invader7 goes.redisd[17309]: net.(*netFD).accept(0xc420204000, 0x0, 0x1208c60, 0xc420222000)
Jan 5 22:04:30 invader7 goes.redisd[17309]: /usr/local/go/src/net/fd_unix.go:419 +0x238
Jan 5 22:04:30 invader7 goes.redisd[17309]: net.(*TCPListener).accept(0xc420136170, 0xc42001d728, 0xc42001d738, 0xc42001d718)
Jan 5 22:04:30 invader7 goes.redisd[17309]: /usr/local/go/src/net/tcpsock_posix.go:132 +0x2e
Jan 5 22:04:30 invader7 goes.redisd[17309]: net.(*TCPListener).Accept(0xc420136170, 0x1244320, 0xc420136170, 0x0, 0x0)
Jan 5 22:04:30 invader7 goes.redisd[17309]: /usr/local/go/src/net/tcpsock.go:222 +0x49
Jan 5 22:04:30 invader7 goes.redisd[17309]: github.com/platinasystems/go/vendor/github.com/platinasystems/go-redis-server.(*Server).Serve(0xc420202000, 0x120dda0, 0xc420136170, 0x0, 0x0)
Jan 5 22:04:30 invader7 goes.redisd[17309]: /home/fyang/gopath/src/github.com/platinasystems/go/vendor/github.com/platinasystems/go-redis-server/server.go:78 +0xb3
Jan 5 22:04:30 invader7 goes.redisd[17309]: github.com/platinasystems/go/vendor/github.com/platinasystems/go-redis-server.(*Server).Start(0xc420202000, 0x0, 0x9)
Jan 5 22:04:30 invader7 goes.redisd[17309]: /home/fyang/gopath/src/github.com/platinasystems/go/vendor/github.com/platinasystems/go-redis-server/server.go:60 +0x3d
Jan 5 22:04:30 invader7 goes.redisd[17309]: created by github.com/platinasystems/go/internal/required/redisd.(*Redisd).listen
Jan 5 22:04:30 invader7 goes.redisd[17309]: /home/fyang/gopath/src/github.com/platinasystems/go/internal/required/redisd/redisd.go:374 +0xc4c
Jan 5 22:04:30 invader7 goes.redisd[17309]:
Jan 5 22:04:30 invader7 goes.redisd[17309]: goroutine 35 [IO wait]:
Jan 5 22:04:30 invader7 goes.redisd[17309]: net.runtime_pollWait(0x7f9a30bfa4d8, 0x72, 0x0)
Jan 5 22:04:30 invader7 goes.redisd[17309]: /usr/local/go/src/runtime/netpoll.go:160 +0x59
Jan 5 22:04:30 invader7 goes.redisd[17309]: net.(*pollDesc).wait(0xc4202040d0, 0x72, 0xc42021a640, 0xc42000c160)
Jan 5 22:04:30 invader7 goes.redisd[17309]: /usr/local/go/src/net/fd_poll_runtime.go:73 +0x38
Jan 5 22:04:30 invader7 goes.redisd[17309]: net.(*pollDesc).waitRead(0xc4202040d0, 0x120a560, 0xc42000c160)
Jan 5 22:04:30 invader7 goes.redisd[17309]: /usr/local/go/src/net/fd_poll_runtime.go:78 +0x34
Jan 5 22:04:30 invader7 goes.redisd[17309]: net.(*netFD).accept(0xc420204070, 0x0, 0x1208c60, 0xc420222020)
Jan 5 22:04:30 invader7 goes.redisd[17309]: /usr/local/go/src/net/fd_unix.go:419 +0x238
Jan 5 22:04:30 invader7 goes.redisd[17309]: net.(*TCPListener).accept(0xc4201362b8, 0xc42021a728, 0xc42021a738, 0xc42021a718)
Jan 5 22:04:30 invader7 goes.redisd[17309]: /usr/local/go/src/net/tcpsock_posix.go:132 +0x2e
Jan 5 22:04:30 invader7 goes.redisd[17309]: net.(*TCPListener).Accept(0xc4201362b8, 0x1244320, 0xc4201362b8, 0x0, 0x0)
Jan 5 22:04:30 invader7 goes.redisd[17309]: /usr/local/go/src/net/tcpsock.go:222 +0x49
Jan 5 22:04:30 invader7 goes.redisd[17309]: github.com/platinasystems/go/vendor/github.com/platinasystems/go-redis-server.(*Server).Serve(0xc4202023c0, 0x120dda0, 0xc4201362b8, 0x0, 0x0)
Jan 5 22:04:30 invader7 goes.redisd[17309]: /home/fyang/gopath/src/github.com/platinasystems/go/vendor/github.com/platinasystems/go-redis-server/server.go:78 +0xb3
Jan 5 22:04:30 invader7 goes.redisd[17309]: github.com/platinasystems/go/vendor/github.com/platinasystems/go-redis-server.(*Server).Start(0xc4202023c0, 0x0, 0x8)
Jan 5 22:04:30 invader7 goes.redisd[17309]: /home/fyang/gopath/src/github.com/platinasystems/go/vendor/github.com/platinasystems/go-redis-server/server.go:60 +0x3d
Jan 5 22:04:30 invader7 goes.redisd[17309]: created by github.com/platinasystems/go/internal/required/redisd.(*Redisd).listen
Jan 5 22:04:30 invader7 goes.redisd[17309]: /home/fyang/gopath/src/github.com/platinasystems/go/internal/required/redisd/redisd.go:374 +0xc4c
Jan 5 22:04:30 invader7 goes.redisd[17309]:
Jan 5 22:04:30 invader7 goes.redisd[17309]: goroutine 36 [IO wait]:
Jan 5 22:04:30 invader7 goes.redisd[17309]: net.runtime_pollWait(0x7f9a30bfa418, 0x72, 0x0)
Jan 5 22:04:30 invader7 goes.redisd[17309]: /usr/local/go/src/runtime/netpoll.go:160 +0x59
Jan 5 22:04:30 invader7 goes.redisd[17309]: net.(*pollDesc).wait(0xc420204140, 0x72, 0xc42021ae40, 0xc42000c160)
Jan 5 22:04:30 invader7 goes.redisd[17309]: /usr/local/go/src/net/fd_poll_runtime.go:73 +0x38
Jan 5 22:04:30 invader7 goes.redisd[17309]: net.(*pollDesc).waitRead(0xc420204140, 0x120a560, 0xc42000c160)
Jan 5 22:04:30 invader7 goes.redisd[17309]: /usr/local/go/src/net/fd_poll_runtime.go:78 +0x34
Jan 5 22:04:30 invader7 goes.redisd[17309]: net.(*netFD).accept(0xc4202040e0, 0x0, 0x1208c60, 0xc4200d3280)
Jan 5 22:04:30 invader7 goes.redisd[17309]: /usr/local/go/src/net/fd_unix.go:419 +0x238
Jan 5 22:04:30 invader7 goes.redisd[17309]: net.(*TCPListener).accept(0xc420136400, 0x4372fe, 0xc42021aef0, 0x6792a8)
Jan 5 22:04:30 invader7 goes.redisd[17309]: /usr/local/go/src/net/tcpsock_posix.go:132 +0x2e
Jan 5 22:04:30 invader7 goes.redisd[17309]: net.(*TCPListener).Accept(0xc420136400, 0xb8e338, 0xc420202780, 0x1212560, 0xc4200241c0)
Jan 5 22:04:30 invader7 goes.redisd[17309]: /usr/local/go/src/net/tcpsock.go:222 +0x49
Jan 5 22:04:30 invader7 goes.redisd[17309]: github.com/platinasystems/go/vendor/github.com/platinasystems/go-redis-server.(*Server).Serve(0xc420202780, 0x120dda0, 0xc420136400, 0x0, 0x0)
Jan 5 22:04:30 invader7 goes.redisd[17309]: /home/fyang/gopath/src/github.com/platinasystems/go/vendor/github.com/platinasystems/go-redis-server/server.go:78 +0xb3
Jan 5 22:04:30 invader7 goes.redisd[17309]: github.com/platinasystems/go/vendor/github.com/platinasystems/go-redis-server.(*Server).Start(0xc420202780, 0x0, 0xf)
Jan 5 22:04:30 invader7 goes.redisd[17309]: /home/fyang/gopath/src/github.com/platinasystems/go/vendor/github.com/platinasystems/go-redis-server/server.go:60 +0x3d
Jan 5 22:04:30 invader7 goes.redisd[17309]: created by github.com/platinasystems/go/internal/required/redisd.(*Redisd).listen
Jan 5 22:04:30 invader7 goes.redisd[17309]: /home/fyang/gopath/src/github.com/platinasystems/go/internal/required/redisd/redisd.go:374 +0xc4c
Jan 5 22:04:30 invader7 goes.redisd[17309]:
Jan 5 22:04:30 invader7 goes.redisd[17309]: goroutine 37 [IO wait]:
Jan 5 22:04:30 invader7 goes.redisd[17309]: net.runtime_pollWait(0x7f9a30bfa358, 0x72, 0x0)
Jan 5 22:04:30 invader7 goes.redisd[17309]: /usr/local/go/src/runtime/netpoll.go:160 +0x59
Jan 5 22:04:30 invader7 goes.redisd[17309]: net.(*pollDesc).wait(0xc4202041b0, 0x72, 0xc42021b640, 0xc42000c160)
Jan 5 22:04:30 invader7 goes.redisd[17309]: /usr/local/go/src/net/fd_poll_runtime.go:73 +0x38
Jan 5 22:04:30 invader7 goes.redisd[17309]: net.(*pollDesc).waitRead(0xc4202041b0, 0x120a560, 0xc42000c160)
Jan 5 22:04:30 invader7 goes.redisd[17309]: /usr/local/go/src/net/fd_poll_runtime.go:78 +0x34
Jan 5 22:04:30 invader7 goes.redisd[17309]: net.(*netFD).accept(0xc420204150, 0x0, 0x1208c60, 0xc420139c80)
Jan 5 22:04:30 invader7 goes.redisd[17309]: /usr/local/go/src/net/fd_unix.go:419 +0x238
Jan 5 22:04:30 invader7 goes.redisd[17309]: net.(*TCPListener).accept(0xc420136548, 0xc42021b728, 0xc42021b738, 0xc42021b718)
Jan 5 22:04:30 invader7 goes.redisd[17309]: /usr/local/go/src/net/tcpsock_posix.go:132 +0x2e
Jan 5 22:04:30 invader7 goes.redisd[17309]: net.(*TCPListener).Accept(0xc420136548, 0x1244320, 0xc420136548, 0x0, 0x0)
Jan 5 22:04:30 invader7 goes.redisd[17309]: /usr/local/go/src/net/tcpsock.go:222 +0x49
Jan 5 22:04:30 invader7 goes.redisd[17309]: github.com/platinasystems/go/vendor/github.com/platinasystems/go-redis-server.(*Server).Serve(0xc420202b40, 0x120dda0, 0xc420136548, 0x0, 0x0)
Jan 5 22:04:30 invader7 goes.redisd[17309]: /home/fyang/gopath/src/github.com/platinasystems/go/vendor/github.com/platinasystems/go-redis-server/server.go:78 +0xb3
Jan 5 22:04:30 invader7 goes.redisd[17309]: github.com/platinasystems/go/vendor/github.com/platinasystems/go-redis-server.(*Server).Start(0xc420202b40, 0x0, 0x1d)
Jan 5 22:04:30 invader7 goes.redisd[17309]: /home/fyang/gopath/src/github.com/platinasystems/go/vendor/github.com/platinasystems/go-redis-server/server.go:60 +0x3d
Jan 5 22:04:30 invader7 goes.redisd[17309]: created by github.com/platinasystems/go/internal/required/redisd.(*Redisd).listen
Jan 5 22:04:30 invader7 goes.redisd[17309]: /home/fyang/gopath/src/github.com/platinasystems/go/internal/required/redisd/redisd.go:374 +0xc4c
Jan 5 22:04:30 invader7 goes.redisd[17309]: exit status 2
Jan 5 22:04:40 invader7 goes.goes-daemons[17304]: (platina,redis.ready) timeout

control-c during "vnet show ip fib" can crash vnet

I'm using bgp_simple to inject over 15000 routes. I issued the show command and did a control-c since it was a lot of output.

stig@invader1:~/bin$ sudo goes vnet show ip fib | wc -l
15504

stig@invader1:~/bin$ sudo goes vnet show ip fib | more
Table Destination Adjacency
0 1.0.0.0/24 6: rewrite eth-0-0 IP4: 02:46:8a:00:02:a2 -> 02:46:8a:00:01:96
0 1.0.4.0/24 6: rewrite eth-0-0 IP4: 02:46:8a:00:02:a2 -> 02:46:8a:00:01:96
0 1.0.5.0/24 6: rewrite eth-0-0 IP4: 02:46:8a:00:02:a2 -> 02:46:8a:00:01:96
0 1.0.6.0/24 6: rewrite eth-0-0 IP4: 02:46:8a:00:02:a2 -> 02:46:8a:00:01:96
0 1.0.7.0/24 6: rewrite eth-0-0 IP4: 02:46:8a:00:02:a2 -> 02:46:8a:00:01:96
0 1.0.38.0/24 6: rewrite eth-0-0 IP4: 02:46:8a:00:02:a2 -> 02:46:8a:00:01:96
0 1.0.64.0/18 6: rewrite eth-0-0 IP4: 02:46:8a:00:02:a2 -> 02:46:8a:00:01:96
0 1.0.128.0/17 6: rewrite eth-0-0 IP4: 02:46:8a:00:02:a2 -> 02:46:8a:00:01:96
0 1.0.128.0/18 6: rewrite eth-0-0 IP4: 02:46:8a:00:02:a2 -> 02:46:8a:00:01:96
0 1.0.128.0/19 6: rewrite eth-0-0 IP4: 02:46:8a:00:02:a2 -> 02:46:8a:00:01:96
0 1.0.128.0/24 6: rewrite eth-0-0 IP4: 02:46:8a:00:02:a2 -> 02:46:8a:00:01:96
0 1.0.129.0/24 6: rewrite eth-0-0 IP4: 02:46:8a:00:02:a2 -> 02:46:8a:00:01:96
0 1.0.131.0/24 6: rewrite eth-0-0 IP4: 02:46:8a:00:02:a2 -> 02:46:8a:00:01:96
0 1.0.139.0/24 6: rewrite eth-0-0 IP4: 02:46:8a:00:02:a2 -> 02:46:8a:00:01:96
0 1.0.142.0/24 6: rewrite eth-0-0 IP4: 02:46:8a:00:02:a2 -> 02:46:8a:00:01:96
0 1.0.160.0/19 6: rewrite eth-0-0 IP4: 02:46:8a:00:02:a2 -> 02:46:8a:00:01:96
0 1.0.160.0/21 6: rewrite eth-0-0 IP4: 02:46:8a:00:02:a2 -> 02:46:8a:00:01:96
0 1.0.192.0/18 6: rewrite eth-0-0 IP4: 02:46:8a:00:02:a2 -> 02:46:8a:00:01:96
0 1.0.192.0/19 6: rewrite eth-0-0 IP4: 02:46:8a:00:02:a2 -> 02:46:8a:00:01:96
0 1.0.192.0/21 6: rewrite eth-0-0 IP4: 02:46:8a:00:02:a2 -> 02:46:8a:00:01:96
0 1.0.208.0/22 6: rewrite eth-0-0 IP4: 02:46:8a:00:02:a2 -> 02:46:8a:00:01:96
^c

stig@invader1:~/bin$ sudo goes vnet show ip fib |more
vnet: dial unix /run/goes/socks/vnet: connect: connection refused

Mar 10 15:02:48 invader1 goes.vnetd[4594]: panic: unix:/run/goes/socks/vnet -> unix:@
Mar 10 15:02:48 invader1 goes.vnetd[4594]:
Mar 10 15:02:48 invader1 goes.vnetd[4594]: goroutine 22 [running]:
Mar 10 15:02:48 invader1 goes.vnetd[4594]: panic(0xed3a80, 0xc820084d88)
Mar 10 15:02:48 invader1 goes.vnetd[4594]: /usr/local/go/src/runtime/panic.go:464 +0x3e6
Mar 10 15:02:48 invader1 goes.vnetd[4594]: github.com/platinasystems/go/elib/socket.(*socket).ErrorReady(0xc820084d88, 0x0, 0x0)
Mar 10 15:02:48 invader1 goes.vnetd[4594]: /home/stig/go/src/github.com/platinasystems/go/elib/socket/socket.go:221 +0x31
Mar 10 15:02:48 invader1 goes.vnetd[4594]: github.com/platinasystems/go/elib/cli.(*File).ErrorReady(0xc820be5338, 0x0, 0x0)
Mar 10 15:02:48 invader1 goes.vnetd[4594]: :59 +0x5c
Mar 10 15:02:48 invader1 goes.vnetd[4594]: github.com/platinasystems/go/elib/iomux.(*Mux).do(0x19078e0, 0xc8236272f0)
Mar 10 15:02:48 invader1 goes.vnetd[4594]: /home/stig/go/src/github.com/platinasystems/go/elib/iomux/epoll.go:153 +0x329
Mar 10 15:02:48 invader1 goes.vnetd[4594]: github.com/platinasystems/go/elib/iomux.(*Mux).EventPoll(0x19078e0)
Mar 10 15:02:48 invader1 goes.vnetd[4594]: /home/stig/go/src/github.com/platinasystems/go/elib/iomux/epoll.go:166 +0x24b
Mar 10 15:02:48 invader1 goes.vnetd[4594]: github.com/platinasystems/go/elib/loop.(*eventLoop).eventPoller(0xc820186370, 0x7f77237cb280, 0x19078e0)
Mar 10 15:02:48 invader1 goes.vnetd[4594]: /home/stig/go/src/github.com/platinasystems/go/elib/loop/event.go:158 +0x27
Mar 10 15:02:48 invader1 goes.vnetd[4594]: created by github.com/platinasystems/go/elib/loop.(*eventLoop).startPoller
Mar 10 15:02:48 invader1 goes.vnetd[4594]: /home/stig/go/src/github.com/platinasystems/go/elib/loop/event.go:161 +0x49
Mar 10 15:02:50 invader1 goes.vnetd[4594]: exit status 2

panic: unknown link info attribute kind SLAVE_KIND

SW Version: a2dff17

Saw this trying to install goes on Invader2. Reproducible 100% of the time.
platina@invader2:$ sudo ./goes-latest install
start: (platina,vnet.ready) timeout
install: start: exit status 1

syslog shows:
Apr 16 21:32:37 invader2 goes.vnetd[28108]: panic: unknown link info attribute kind SLAVE_KIND
Apr 16 21:32:37 invader2 goes.vnetd[28108]:
Apr 16 21:32:37 invader2 goes.vnetd[28108]: goroutine 25 [running]:
Apr 16 21:32:37 invader2 goes.vnetd[28108]: panic(0xa4aa00, 0xc4207ecec0)
Apr 16 21:32:37 invader2 goes.vnetd[28108]: /usr/local/go/src/runtime/panic.go:500 +0x1a1
Apr 16 21:32:37 invader2 goes.vnetd[28108]: github.com/platinasystems/go/internal/netlink.parse_link_info(0xc4202177bc, 0xe8, 0xc844, 0xc420bf6fc0)
Apr 16 21:32:37 invader2 goes.vnetd[28108]: /home/fyang/gopath/src/github.com/platinasystems/go/internal/netlink/attr.go:217 +0x3b4
Apr 16 21:32:37 invader2 goes.vnetd[28108]: github.com/platinasystems/go/internal/netlink.(*IfInfoMessage).Write(0xc420be5500, 0xc4202175a0, 0x5d8, 0xca60, 0x4e0, 0x0, 0x0)
Apr 16 21:32:37 invader2 goes.vnetd[28108]: /home/fyang/gopath/src/github.com/platinasystems/go/internal/netlink/msg.go:430 +0x4e8
Apr 16 21:32:37 invader2 goes.vnetd[28108]: github.com/platinasystems/go/internal/netlink.(*Socket).gorx(0xc4201ff400)
Apr 16 21:32:37 invader2 goes.vnetd[28108]: /home/fyang/gopath/src/github.com/platinasystems/go/internal/netlink/socket.go:338 +0x34a
Apr 16 21:32:37 invader2 goes.vnetd[28108]: created by github.com/platinasystems/go/internal/netlink.NewWithConfig
Apr 16 21:32:37 invader2 goes.vnetd[28108]: /home/fyang/gopath/src/github.com/platinasystems/go/internal/netlink/socket.go:199 +0x646
Apr 16 21:32:39 invader2 goes.vnetd[28108]: exit status 2

Same version of SW installed OK on invader7, so looks that there may be some dependencies.

Punt path is broken in invader TX direction

Version:

donn@invader8:~/workspace/gitlab.com/donnlee/tor/invader/testing/punt_path$ /usr/bin/goes version
71edbc1d0d43cdeb99cd618db30d0cb33d8ef355

which is:

commit 71edbc1d0d43cdeb99cd618db30d0cb33d8ef355
Author: Tom Grennan <[email protected]>
Date:   Fri Dec 30 17:34:31 2016 -0800

Test setup is back-to-back i8 and i14, 100Gb copper cable. i14 is running pktgen:
i8:eth-29-0-------100g------i14:eth1.129

I have a script that does a scp transfer of a 500MB .iso file between them and loops forever.
i8 initiates the scp transfers and downloads .iso file from i14.

scp loop cmd:

donn@invader8:~/workspace/gitlab.com/donnlee/tor/invader/testing/punt_path$ while ./scp_loop.py : ; do sleep 1; done > ~/scp_loop_$(date "+%Y_%m_%d_%H%M").log 2>&1

My script starts at:

Starting file xfer test at Fri Jan  6 11:14:24 PST 2017:
Transferred: sent 98980, received 548027552 bytes, in 2.0 seconds
Bytes per second: sent 48616.1, received 269175072.6
Done with file xfer at Fri Jan  6 11:14:26 PST 2017.

About 13 hours later, the scp file transfer fails starting at: Sat Jan 7 00:29:30 PST 2017

Starting file xfer test at Sat Jan  7 00:29:24 PST 2017:
Transferred: sent 98900, received 548027552 bytes, in 1.7 seconds
Bytes per second: sent 59514.7, received 329784662.9
Done with file xfer at Sat Jan  7 00:29:26 PST 2017.

Starting file xfer test at Sat Jan  7 00:29:27 PST 2017:
Transferred: sent 98920, received 548027552 bytes, in 1.6 seconds
Bytes per second: sent 61817.6, received 342476312.0
Done with file xfer at Sat Jan  7 00:29:29 PST 2017.

Starting file xfer test at Sat Jan  7 00:29:30 PST 2017:
Done with file xfer at Sat Jan  7 00:45:13 PST 2017.

Starting file xfer test at Sat Jan  7 00:45:14 PST 2017:
Done with file xfer at Sat Jan  7 00:45:17 PST 2017.

Starting file xfer test at Sat Jan  7 00:45:18 PST 2017:
Done with file xfer at Sat Jan  7 00:45:20 PST 2017.

Starting file xfer test at Sat Jan  7 00:45:21 PST 2017:
Done with file xfer at Sat Jan  7 00:45:23 PST 2017.

Starting file xfer test at Sat Jan  7 00:45:24 PST 2017:
Done with file xfer at Sat Jan  7 00:45:26 PST 2017.

Nothing in syslog at this time.

vnet is alive:

donn@invader8:~/workspace/go/src/github.com/platinasystems/go$ sudo goes vnet sh ip fib
[sudo] password for donn:
 Table                   Destination           Adjacency
     0                  10.0.29.0/24  1: glean eth-29-0
     0                  10.0.29.2/32  4: rewrite eth-29-0 IP4: 02:46:8a:00:05:ac -> 02:46:8a:00:06:48
     0                  10.0.29.8/32  2: local eth-29-0
                                        tx pipe packets      10367949796
                                        tx pipe bytes     15307058513083

Investigation: Have i8 try to ping i14.

Symptom: i8 cannot reach i14 anymore:

donn@invader8:~/workspace/gitlab.com/donnlee/tor/invader/testing/punt_path$ ping 10.0.29.14
PING 10.0.29.14 (10.0.29.14) 56(84) bytes of data.
^C
--- 10.0.29.14 ping statistics ---
2 packets transmitted, 0 received, 100% packet loss, time 1000ms

Symptom: i8 cannot resolve arp for i14 (10.0.29.14):

donn@invader8:~/workspace/gitlab.com/donnlee/tor/invader/testing/punt_path$ sudo arp -n
Address                  HWtype  HWaddress           Flags Mask            Iface
192.168.101.2            ether   00:17:cb:80:33:4c   C                     eth0
10.0.29.14                       (incomplete)                              eth-29-0

i14 sees the ARP request from i8 and sends a reply:

root@invader14-pktgen:~/SWITCH/sdkapp# tcpdump -nni eth1.129 arp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth1.129, link-type EN10MB (Ethernet), capture size 65535 bytes
11:40:59.856035 ARP, Request who-has 10.0.29.14 tell 10.0.29.8, length 50
11:40:59.856053 ARP, Reply 10.0.29.14 is-at 02:46:8a:00:06:48, length 28
11:41:00.853049 ARP, Request who-has 10.0.29.14 tell 10.0.29.8, length 50
11:41:00.853063 ARP, Reply 10.0.29.14 is-at 02:46:8a:00:06:48, length 28
11:41:01.857032 ARP, Request who-has 10.0.29.14 tell 10.0.29.8, length 50
11:41:01.857046 ARP, Reply 10.0.29.14 is-at 02:46:8a:00:06:48, length 28
11:41:02.862148 ARP, Request who-has 10.0.29.14 tell 10.0.29.8, length 50
11:41:02.862162 ARP, Reply 10.0.29.14 is-at 02:46:8a:00:06:48, length 28
11:41:03.860988 ARP, Request who-has 10.0.29.14 tell 10.0.29.8, length 50
11:41:03.861000 ARP, Reply 10.0.29.14 is-at 02:46:8a:00:06:48, length 28
11:41:04.861036 ARP, Request who-has 10.0.29.14 tell 10.0.29.8, length 50
11:41:04.861048 ARP, Reply 10.0.29.14 is-at 02:46:8a:00:06:48, length 28

But reply is never received on i8 x86.

This is i8's view of the ping/arp:

donn@invader8:~/workspace/gitlab.com/donnlee/tor/invader/testing/punt_path$ sudo tcpdump -nni eth-29-0 arp
[sudo] password for donn:
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth-29-0, link-type EN10MB (Ethernet), capture size 262144 bytes
11:40:59.738762 ARP, Request who-has 10.0.29.14 tell 10.0.29.8, length 28
11:41:00.735729 ARP, Request who-has 10.0.29.14 tell 10.0.29.8, length 28
11:41:01.739704 ARP, Request who-has 10.0.29.14 tell 10.0.29.8, length 28
11:41:02.744829 ARP, Request who-has 10.0.29.14 tell 10.0.29.8, length 28

This tells me the path from i14 to i8 is broken, and I suspect (but have no experience to prove) the arp-reply is dropped in the i8 TH or between i8 TH and i8 x86 punt path.

I've left i8 in this state so a goes coder can do brain surgery on it.
Anyone want to look at this while I have it in this failed state?

index out of range on goes start

Getting "index out of range" error in syslog on goes start. vnetd wouldn't launch. Doesn't happen all the time.

version: 6fa4a9f

12300 ? Ssl 0:00 goes-daemons
12306 ? Sl 0:00 _ redisd
12321 ? Sl 0:00 _ uptimed
12322 ? Sl 0:00 _ nld

/var/log/syslog:
...
Feb 4 15:13:16 invader17 goes.vnetd[12326]: runtime error: index out of range: goroutine 36 [running]:
Feb 4 15:13:16 invader17 goes.vnetd[12326]: runtime/debug.Stack(0xc420046610, 0xa46000, 0xc42000c0a0)
Feb 4 15:13:16 invader17 goes.vnetd[12326]: /usr/local/go/src/runtime/debug/stack.go:24 +0x79
Feb 4 15:13:16 invader17 goes.vnetd[12326]: github.com/platinasystems/go/elib/loop.(*Loop).doEvent.func1(0xc420194000)
Feb 4 15:13:16 invader17 goes.vnetd[12326]: /home/fyang/gopath/src/github.com/platinasystems/go/elib/loop/event.go:128 +0x91
Feb 4 15:13:16 invader17 goes.vnetd[12326]: panic(0xa46000, 0xc42000c0a0)
Feb 4 15:13:16 invader17 goes.vnetd[12326]: /usr/local/go/src/runtime/panic.go:458 +0x243
Feb 4 15:13:16 invader17 goes.vnetd[12326]: github.com/platinasystems/go/vnet/devices/ethernet/switch/fe1/internal/fe1a.(*pipe_counter_eviction_fifo_elt).evict_pool
_counter(0xc420046768, 0xc420218000, 0xc42022c180, 0x0)
Feb 4 15:13:16 invader17 goes.vnetd[12326]: /home/fyang/gopath/src/github.com/platinasystems/go/vnet/devices/ethernet/switch/fe1/internal/fe1a/pipe_counters
.go:585 +0xcd
Feb 4 15:13:16 invader17 goes.vnetd[12326]: github.com/platinasystems/go/vnet/devices/ethernet/switch/fe1/internal/fe1a.(*fe1a).decode_eviction_fifo(0xc420218000, 0
xc420ed8138, 0x3, 0x3fb2)
Feb 4 15:13:16 invader17 goes.vnetd[12326]: /home/fyang/gopath/src/github.com/platinasystems/go/vnet/devices/ethernet/switch/fe1/internal/fe1a/pipe_counters
.go:660 +0x32c
Feb 4 15:13:16 invader17 goes.vnetd[12326]: github.com/platinasystems/go/vnet/devices/ethernet/switch/fe1/internal/fe1a.(*fe1a).handle_fifo_data(0xc420218000, 0xc42
0ed8000, 0x2eec, 0x4000, 0xc4203eb380)
Feb 4 15:13:16 invader17 goes.vnetd[12326]: /home/fyang/gopath/src/github.com/platinasystems/go/vnet/devices/ethernet/switch/fe1/internal/fe1a/pipe_counters
.go:330 +0xaa
Feb 4 15:13:16 invader17 goes.vnetd[12326]: github.com/platinasystems/go/vnet/devices/ethernet/switch/fe1/internal/fe1a.(*fe1a).pipe_counter_eviction_fifo_sync(0xc4
20218000)
Feb 4 15:13:16 invader17 goes.vnetd[12326]: /home/fyang/gopath/src/github.com/platinasystems/go/vnet/devices/ethernet/switch/fe1/internal/fe1a/pipe_counters
.go:343 +0x75
Feb 4 15:13:16 invader17 goes.vnetd[12326]: github.com/platinasystems/go/vnet/devices/ethernet/switch/fe1/internal/fe1a.(*fe1a).get_port_counters_helper(0xc42021800
0, 0xc42035de60, 0xc4209d8a20, 0xc420040000)
Feb 4 15:13:16 invader17 goes.vnetd[12326]: /home/fyang/gopath/src/github.com/platinasystems/go/vnet/devices/ethernet/switch/fe1/internal/fe1a/port_counters
.go:308 +0x764
Feb 4 15:13:16 invader17 goes.vnetd[12326]: github.com/platinasystems/go/vnet/devices/ethernet/switch/fe1/internal/fe1a.(*fe1a).get_port_counters(0xc420218000, 0xc4
2035de60, 0xc4209d8a20)
Feb 4 15:13:16 invader17 goes.vnetd[12326]: /home/fyang/gopath/src/github.com/platinasystems/go/vnet/devices/ethernet/switch/fe1/internal/fe1a/port_counters
.go:153 +0x46
Feb 4 15:13:16 invader17 goes.vnetd[12326]: github.com/platinasystems/go/vnet/devices/ethernet/switch/fe1/internal/fe1a.(*Port).GetHwInterfaceCounterValues(0xc42035
de60, 0xc4209d8a20)
Feb 4 15:13:16 invader17 goes.vnetd[12326]: /home/fyang/gopath/src/github.com/platinasystems/go/vnet/devices/ethernet/switch/fe1/internal/fe1a/port_init.go:
158 +0x5d
Feb 4 15:13:16 invader17 goes.vnetd[12326]: github.com/platinasystems/go/vnet.(*interfaceMain).foreachHwIfCounter(0xc4201946f8, 0x752035e001, 0xc420047e80)
Feb 4 15:13:16 invader17 goes.vnetd[12326]: /home/fyang/gopath/src/github.com/platinasystems/go/vnet/interface_counter.go:233 +0xd3
Feb 4 15:13:16 invader17 goes.vnetd[12326]: github.com/platinasystems/go/vnet.(*Vnet).ForeachHwIfCounter(0xc420194000, 0x4014000000000101, 0xc420047ee0)
Feb 4 15:13:16 invader17 goes.vnetd[12326]: /home/fyang/gopath/src/github.com/platinasystems/go/vnet/interface_counter.go:258 +0x12d
Feb 4 15:13:16 invader17 goes.vnetd[12326]: github.com/platinasystems/go/internal/optional/vnetd.(*ifStatsPoller).EventAction(0xc420194a30)
Feb 4 15:13:16 invader17 goes.vnetd[12326]: /home/fyang/gopath/src/github.com/platinasystems/go/internal/optional/vnetd/vnetd.go:286 +0x8d
Feb 4 15:13:16 invader17 goes.vnetd[12326]: github.com/platinasystems/go/elib/loop.(*loopEvent).do(0xc42014f980)
Feb 4 15:13:16 invader17 goes.vnetd[12326]: /home/fyang/gopath/src/github.com/platinasystems/go/elib/loop/event.go:118 +0x34
Feb 4 15:13:16 invader17 goes.vnetd[12326]: github.com/platinasystems/go/elib/loop.(*Loop).doEvent(0xc420194000, 0xc42014f980)
Feb 4 15:13:16 invader17 goes.vnetd[12326]: /home/fyang/gopath/src/github.com/platinasystems/go/elib/loop/event.go:133 +0x51
Feb 4 15:13:16 invader17 goes.vnetd[12326]: github.com/platinasystems/go/elib/loop.(*Loop).eventHandler(0xc420194000, 0x12320e0, 0xc420194548)
Feb 4 15:13:16 invader17 goes.vnetd[12326]: /home/fyang/gopath/src/github.com/platinasystems/go/elib/loop/event.go:140 +0x86
Feb 4 15:13:16 invader17 goes.vnetd[12326]: created by github.com/platinasystems/go/elib/loop.(*Loop).startHandler
Feb 4 15:13:16 invader17 goes.vnetd[12326]: /home/fyang/gopath/src/github.com/platinasystems/go/elib/loop/event.go:153 +0x124
Feb 4 15:13:19 invader17 goes.vnetd[12326]: done

redis link status out of sync with real time

Saw one time after goes start where redis link status says link is down, but actual link is up.

platina@invader16:/home/fyang$ goes hget platina vnet.eth-7-1.link
false

platina@invader16:/home/fyang$ goes vnet show fe1 port-status phy eth-7-1
Name Live Link Signal Detect Pmd Lock Sigdet Sts Speed Autonegotiate Cl72
eth-7-1:0 true true true 0x211 100G X4 CR done ready
eth-7-1:1 false true true 0x211 100G X4 CR ready
eth-7-1:2 false true true 0x211 100G X4 CR ready
eth-7-1:3 false true true 0x211 100G X4 CR ready

Traffic was passing OK bidirectionally through this link so the real time status is correct.
Looks like a link state change interrupt may have been dropped.
Ports was in auto mode. Set speed on the link to auto again brought the redis and real time link state back in sync.

invalid memory address or nil pointer dereference" after 'goes start'

While attempting to repro #30 , after 'goes start' vnet never responded to 'sho ip fib' and I saw this in syslog:

Mar 23 10:54:17 invader16 goes.vnetd[17471]: runtime error: invalid memory address or nil pointer dereference: goroutine 36 [running]:
Mar 23 10:54:17 invader16 goes.vnetd[17471]: runtime/debug.Stack(0xc420ca5a70, 0xa7c740, 0xc42000c060)
Mar 23 10:54:17 invader16 goes.vnetd[17471]:         /usr/local/go/src/runtime/debug/stack.go:24 +0x79
Mar 23 10:54:17 invader16 goes.vnetd[17471]: github.com/platinasystems/go/elib/loop.(*Loop).doEvent.func1(0xc420186000)
Mar 23 10:54:17 invader16 goes.vnetd[17471]:         /home/donn/workspace/go/src/github.com/platinasystems/go/elib/loop/event.go:128 +0x91
Mar 23 10:54:17 invader16 goes.vnetd[17471]: panic(0xa7c740, 0xc42000c060)
Mar 23 10:54:17 invader16 goes.vnetd[17471]:         /usr/local/go/src/runtime/panic.go:458 +0x243
Mar 23 10:54:17 invader16 goes.vnetd[17471]: github.com/platinasystems/go/vnet/ip.(*Main).AddDelNextHop(0xc42007c168, 0x100000017, 0x100000005, 0xc420af1c00)
Mar 23 10:54:17 invader16 goes.vnetd[17471]:         /home/donn/workspace/go/src/github.com/platinasystems/go/vnet/ip/adjacency.go:459 +0x4c8
Mar 23 10:54:17 invader16 goes.vnetd[17471]: github.com/platinasystems/go/vnet/ip4.(*Main).AddDelRouteNextHop(0xc42007c100, 0xc4212c6528, 0xc4212c6ef0, 0xc4208b9401, 0x50e02000a, 0x1)
Mar 23 10:54:17 invader16 goes.vnetd[17471]:         /home/donn/workspace/go/src/github.com/platinasystems/go/vnet/ip4/fib.go:429 +0x1d8
Mar 23 10:54:17 invader16 goes.vnetd[17471]: github.com/platinasystems/go/vnet/unix.(*Main).ip4RouteMsg(0xc4200d7200, 0xc42127e820, 0xc420ca5e01, 0x42f7ce, 0xbece50)
Mar 23 10:54:17 invader16 goes.vnetd[17471]:         /home/donn/workspace/go/src/github.com/platinasystems/go/vnet/unix/netlink.go:489 +0x1f2
Mar 23 10:54:17 invader16 goes.vnetd[17471]: github.com/platinasystems/go/vnet/unix.(*netlinkEvent).EventAction(0xc421153530)
Mar 23 10:54:17 invader16 goes.vnetd[17471]:         /home/donn/workspace/go/src/github.com/platinasystems/go/vnet/unix/netlink.go:301 +0x466
Mar 23 10:54:17 invader16 goes.vnetd[17471]: github.com/platinasystems/go/elib/loop.(*loopEvent).do(0xc421153560)
Mar 23 10:54:17 invader16 goes.vnetd[17471]:         /home/donn/workspace/go/src/github.com/platinasystems/go/elib/loop/event.go:118 +0x34
Mar 23 10:54:17 invader16 goes.vnetd[17471]: github.com/platinasystems/go/elib/loop.(*Loop).doEvent(0xc420186000, 0xc421153560)
Mar 23 10:54:17 invader16 goes.vnetd[17471]:         /home/donn/workspace/go/src/github.com/platinasystems/go/elib/loop/event.go:133 +0x51
Mar 23 10:54:17 invader16 goes.vnetd[17471]: github.com/platinasystems/go/elib/loop.(*Loop).eventHandler(0xc420186000, 0x128dee0, 0xc420186548)
Mar 23 10:54:17 invader16 goes.vnetd[17471]:         /home/donn/workspace/go/src/github.com/platinasystems/go/elib/loop/event.go:140 +0x86
Mar 23 10:54:17 invader16 goes.vnetd[17471]: created by github.com/platinasystems/go/elib/loop.(*Loop).startHandler
Mar 23 10:54:17 invader16 goes.vnetd[17471]:         /home/donn/workspace/go/src/github.com/platinasystems/go/elib/loop/event.go:153 +0x124

root@invader16:~# /usr/bin/goes version
7d6b2baf82c482a043ce4d875b91fefbbf344c79

commit 7d6b2baf82c482a043ce4d875b91fefbbf344c79
Author: Jason Pang <[email protected]>
Date:   Wed Mar 22 21:14:58 2017 -0700

    Adding non-volatile power logging functions to BMC

master branch of fe1 crash on beta hw but not alpha hw

In tuntap mode I get a crash on invader24 (beta hw) but not on invader1 (alpha hw).

panic: add-next fe1-rx-vlan-redirect: unknown next meth-0

goroutine 1 [running]:
panic(0xd70840, 0xc8204c6a60)
/usr/local/go/src/runtime/panic.go:464 +0x3e6
github.com/platinasystems/go/elib/loop.(*Loop).graphInit(0xc820186000)
/home/stig/go/src/github.com/platinasystems/go/elib/loop/call.go:280 +0x1cd
github.com/platinasystems/go/elib/loop.(*Loop).Run(0xc820186000)
/home/stig/go/src/github.com/platinasystems/go/elib/loop/loop.go:444 +0x13a
github.com/platinasystems/go/vnet.(*Vnet).Run(0xc820186000, 0xc8201560e0, 0x0, 0x0)
/home/stig/go/src/github.com/platinasystems/go/vnet/vnet.go:62 +0xaa
github.com/platinasystems/go/internal/goes/cmd/vnetd.(*cmd).Main(0xc820186000, 0xc82006a160, 0x0, 0x0, 0x0, 0x0)
/home/stig/go/src/github.com/platinasystems/go/internal/goes/cmd/vnetd/vnetd.go:116 +0x876
github.com/platinasystems/go/internal/goes.(mainer).Main-fm(0xc82006a160, 0x0, 0x0, 0x0, 0x0)
/home/stig/go/src/github.com/platinasystems/go/internal/goes/goes.go:223 +0x5e
github.com/platinasystems/go/internal/goes.ByName.Main(0xc82013e540, 0xc82006a160, 0x0, 0x0, 0x0, 0x0)
/home/stig/go/src/github.com/platinasystems/go/internal/goes/goes.go:192 +0xce6
main.main()
/home/stig/go/src/github.com/platinasystems/go/main/goes-platina-mk1/main.go:14 +0x40

meth flapping on older kernel

Using older kernel that doesn't have SRIOV, meth ports are flapping continuously

platina@invader2:~$ goes hget platina version
fe1252f

platina@invader2:~$ uname -r
4.6.0-rc2-platina-mk1-amd64

platina@invader2:~$ goes hdelta
vnet.meth-1.admin: false
vnet.meth-0.admin: true
vnet.meth-1.admin: true
vnet.meth-0.admin: false
vnet.meth-1.admin: false
vnet.meth-0.admin: true
vnet.meth-1.admin: true
vnet.meth-0.admin: false
vnet.meth-1.admin: false
vnet.meth-0.admin: true
vnet.meth-1.admin: true
vnet.meth-0.admin: false
vnet.meth-1.admin: false
vnet.meth-0.admin: true
vnet.meth-1.admin: true
vnet.meth-0.admin: false

'vnet sho ip fib' hangs, punt path broken after GoBGP advertisements.

Pulled go and fe1 tree this afternoon:

donn@invader8:~/workspace/gitlab.com/donnlee/tor/invader/testing/punt_path$ /usr/bin/goes version
12f9ee6ad91dfbb4d7f867b2b19cc68b3eecb2b1

Repro steps:

  1. Establish gobgp session between i8 & i14.
  2. i14 advertises 4K routes to i8 (Donn's go util).
  3. i14 withdraws same 4K routes.
  4. i14 advertises same 4K routes.

vnet sho ip fib hangs indefinitely:

donn@invader8:~/workspace/go/src/github.com/platinasystems/go$ goes vnet sho ip f
<hangs here indefinitely>

vnet is still present in ps:

donn@invader8:~/workspace/gitlab.com/donnlee/tor/invader/testing/punt_path$ ps -e f | tail
18783 ?        SLl  123:22  \_ collectd -C /etc/collectd/collectd.conf -f
 8925 ?        Ss     0:00 ssh-agent
14713 ?        Ssl    0:00 goes-daemons
14721 ?        Sl     2:30  \_ redisd
14752 ?        Sl     0:22  \_ vnetd
14753 ?        Sl     0:08  \_ i2cd
14756 ?        Sl     0:00  \_ qsfp
14761 ?        Sl     0:02  \_ nld
14770 ?        Sl     0:00  \_ uptimed
14775 ?        Sl     0:01  \_ qsfpio

BGP sessions are now broken:

donn@invader8:~/workspace/gitlab.com/donnlee/tor/invader/testing/punt_path$ gobgp neighbor
Peer             AS  Up/Down State       |#Advertised Received Accepted
10.0.29.14    65201 00:00:41 Active      |          0        0        0
192.168.155.2 65202 00:01:04 Active      |          0        0        0

Can no longer ping neighbor:

donn@invader8:~/workspace/gitlab.com/donnlee/tor/invader/testing/punt_path$ ping 10.0.29.14
PING 10.0.29.14 (10.0.29.14) 56(84) bytes of data.
From 10.0.29.8 icmp_seq=1 Destination Host Unreachable
From 10.0.29.8 icmp_seq=2 Destination Host Unreachable
From 10.0.29.8 icmp_seq=3 Destination Host Unreachable
^C
--- 10.0.29.14 ping statistics ---
4 packets transmitted, 0 received, +3 errors, 100% packet loss, time 3001ms

Syslog is clean.

To repair: Must kill and restart all goes daemons.

redis not up during goes start

Occasionally, redis.ready does not get asserted on goes start. goes start fails with when checking vnet.ready = true; but failure is due to redis not up (redis.ready never asserted)

When fail:

In, redisd.go,

func Main(args …string):

…

err = Hook(pub) returns err “read: no such device or address”

…

cpu hangs after repeated goes stop/start

After 2+ hours of repeated goes stop/start loops, cpu hangs
Need power cycle to recover

tested on commit 73003a9

console shows:

BUG: unable to handle kernel NULL pointer dereference at 0000000000000008
IP: [<ffffffffa0608c78>] ixgbevf_alloc_rx_buffers+0x88/0x1f0 [ixgbevf]
PGD 0
Oops: 0000 [#1] SMP
Modules linked in: xt_nat ipt_MASQUERADE nf_nat_masquerade_ipv4 ixgbevf iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype iptable_filter xt_conntrack nf_nat nf_conntrack br_netfilter bridge stp llc overlay iptable_raw nls_utf8 nls_cp437 vfat fat kvm_intel kvm uio_pci_dma i2c_i801 i2c_smbus autofs4 dm_mod ixgbe mdio
CPU: 7 PID: 8609 Comm: kworker/u16:2 Tainted: G W 4.9.8-platina-mk1-amd64 #2
Hardware name: Intel Camelback Mountain Platina DC/Camelback Mountain Platina DC, BIOS 6a724f1-dirty 03/14/2017
Workqueue: ixgbevf ixgbevf_service_task [ixgbevf]
task: ffff88046b0d0000 task.stack: ffffc90006f5c000
RIP: 0010:[<ffffffffa0608c78>] [<ffffffffa0608c78>] ixgbevf_alloc_rx_buffers+0x88/0x1f0 [ixgbevf]
RSP: 0018:ffffc90006f5fd28 EFLAGS: 00010287
RAX: 0000000000000200 RBX: 0000000000000000 RCX: 0000000000000000
RDX: 0000000000000000 RSI: 00000000000001ff RDI: ffff88044e1f9000
RBP: ffffc90006f5fd60 R08: 0000000000000000 R09: 0000000000100000
R10: 00000001006cecce R11: 0000000000000001 R12: 0000000000000000
R13: 00000000fffffe00 R14: 00000000000001ff R15: ffff88044e1f9000
FS: 0000000000000000(0000) GS:ffff88047fdc0000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000008 CR3: 000000007ea07000 CR4: 00000000003406e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Stack:
0000000000000000 ffff880460919b28 ffff880460919840 0000000000000009
ffff880460919b28 ffff88044e1f9000 0000000000001028 ffffc90006f5fdc0
ffffffffa060b1d3 ffff880400001028 0000000000000000 ffff880460919ab0
Call Trace:
[<ffffffffa060b1d3>] ixgbevf_configure+0x5a3/0x710 [ixgbevf]
[<ffffffffa060cd18>] ixgbevf_reinit_locked+0x38/0x70 [ixgbevf]
[<ffffffffa060dd76>] ixgbevf_service_task+0x1a6/0x330 [ixgbevf]
[<ffffffff81091c0a>] process_one_work+0x12a/0x330
[<ffffffff81091e71>] worker_thread+0x61/0x490
[<ffffffff81091e10>] ? process_one_work+0x330/0x330
[<ffffffff81097107>] kthread+0xd7/0xf0
[<ffffffff81097030>] ? kthread_park+0x60/0x60
[<ffffffff814c6c92>] ret_from_fork+0x22/0x30
Code: 49 83 c4 10 48 83 c3 18 49 89 44 24 f0 41 83 c5 01 0f 84 14 01 00 00 66 41 83 ee 01 49 c7 44 24 08 00 00 00 00 0f 84 c8 00 00 00 <48> 83 7b 08 00 75 c5 31 c9 31 f6 48 c7 c2 00 0d 87 81 bf 20 63
RIP [<ffffffffa0608c78>] ixgbevf_alloc_rx_buffers+0x88/0x1f0 [ixgbevf]
RSP <ffffc90006f5fd28>
CR2: 0000000000000008
---[ end trace 41feb073b5b2881a ]---
BUG: unable to handle kernel NULL pointer dereference at 0000000000000009
IP: [<ffffffff810b0e06>] __wake_up_common+0x26/0x80
PGD 0
Oops: 0000 [#2] SMP
Modules linked in: xt_nat ipt_MASQUERADE nf_nat_masquerade_ipv4 ixgbevf iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype iptable_filter xt_conntrack nf_nat nf_conntrack br_netfilter bridge stp llc overlay iptable_raw nls_utf8 nls_cp437 vfat fat kvm_intel kvm uio_pci_dma i2c_i801 i2c_smbus autofs4 dm_mod ixgbe mdio
CPU: 7 PID: 8609 Comm: kworker/u16:2 Tainted: G D W 4.9.8-platina-mk1-amd64 #2
Hardware name: Intel Camelback Mountain Platina DC/Camelback Mountain Platina DC, BIOS 6a724f1-dirty 03/14/2017
task: ffff88046b0d0000 task.stack: ffffc90006f5c000
RIP: 0010:[<ffffffff810b0e06>] [<ffffffff810b0e06>] __wake_up_common+0x26/0x80
RSP: 0018:ffffc90006f5fe50 EFLAGS: 00010086
RAX: 0000000000000082 RBX: ffffc90006f5ff18 RCX: 0000000000000000
RDX: 0000000000000009 RSI: 0000000000000003 RDI: ffffc90006f5ff18
RBP: ffffc90006f5fe88 R08: 0000000000000000 R09: 0000000000000000
R10: 000000000007e952 R11: 000000000007e952 R12: ffffc90006f5ff20
R13: 0000000000000082 R14: 0000000000000000 R15: 0000000000000003
FS: 0000000000000000(0000) GS:ffff88047fdc0000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000028 CR3: 000000007ea07000 CR4: 00000000003406e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Stack:
0000000100000046 0000000000000000 ffffc90006f5ff18 ffffc90006f5ff10
0000000000000082 0000000000000000 0000000000000000 ffffc90006f5fe98
ffffffff810b0e6e ffffc90006f5fec0 ffffffff810b17e2 0000000000000000
Call Trace:
[<ffffffff810b0e6e>] __wake_up_locked+0xe/0x10
[<ffffffff810b17e2>] complete+0x32/0x50
[<ffffffff8107af5e>] mm_release+0x9e/0x100
[<ffffffff81080814>] do_exit+0x284/0xab0
[<ffffffff814c8417>] rewind_stack_do_exit+0x17/0x19
Code: 00 00 00 00 00 55 48 89 e5 41 57 41 56 41 55 41 54 41 89 f7 53 4c 8d 67 08 41 89 ce 48 83 ec 10 89 55 cc 48 8b 57 08 4c 89 45 d0 <48> 8b 32 49 39 d4 48 8d 42 e8 4c 8d 6e e8 75 05 eb 38 49 89 d5
RIP [<ffffffff810b0e06>] __wake_up_common+0x26/0x80
RSP <ffffc90006f5fe50>
CR2: 0000000000000009
---[ end trace 41feb073b5b2881b ]---
Fixing recursive fault but reboot is needed!
BUG: unable to handle kernel paging request at ffffffffffffffd8
IP: [<ffffffff81097a2b>] kthread_data+0xb/0x20
PGD 7ea08067 PUD 7ea0a067
PMD 0
Oops: 0000 [#3] SMP
Modules linked in: xt_nat ipt_MASQUERADE nf_nat_masquerade_ipv4 ixgbevf iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype iptable_filter xt_conntrack nf_nat nf_conntrack br_netfilter bridge stp llc overlay iptable_raw nls_utf8 nls_cp437 vfat fat kvm_intel kvm uio_pci_dma i2c_i801 i2c_smbus autofs4 dm_mod ixgbe mdio
CPU: 7 PID: 8609 Comm: kworker/u16:2 Tainted: G D W 4.9.8-platina-mk1-amd64 #2
Hardware name: Intel Camelback Mountain Platina DC/Camelback Mountain Platina DC, BIOS 6a724f1-dirty 03/14/2017
task: ffff88046b0d0000 task.stack: ffffc90006f5c000
RIP: 0010:[<ffffffff81097a2b>] [<ffffffff81097a2b>] kthread_data+0xb/0x20
RSP: 0018:ffffc90006f5fe78 EFLAGS: 00010002
RAX: 0000000000000000 RBX: 0000000000015300 RCX: 0000000000000007
RDX: ffff88046d805000 RSI: ffff88046b0d0000 RDI: ffff88046b0d0000
RBP: ffffc90006f5fe78 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000001 R11: 00000000000001da R12: ffff88047fdd5300
R13: ffff88046b0d0000 R14: ffff88046b0d04b0 R15: 0000000000000000
FS: 0000000000000000(0000) GS:ffff88047fdc0000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000028 CR3: 000000007ea07000 CR4: 00000000003406e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Stack:
ffffc90006f5fe88 ffffffff81092a79 ffffc90006f5fec8 ffffffff814c328e
ffffc90006f5fef0 ffff88046b0d0000 0000000000000009 0000000000000046
ffff88046b0d0000 0000000000000000 ffffc90006f5fee0 ffffffff814c3571
Call Trace:
[<ffffffff81092a79>] wq_worker_sleeping+0x9/0x80
[<ffffffff814c328e>] __schedule+0x2ce/0x580
[<ffffffff814c3571>] schedule+0x31/0x80
[<ffffffff81080e14>] do_exit+0x884/0xab0
[<ffffffff814c8417>] rewind_stack_do_exit+0x17/0x19
Code: be 7d 04 00 00 48 c7 c7 b8 75 6d 81 e8 af 5c fe ff eb ca 66 66 66 66 2e 0f 1f 84 00 00 00 00 00 48 8b 87 50 04 00 00 55 48 89 e5 <48> 8b 40 d8 5d c3 66 66 66 66 66 66 2e 0f 1f 84 00 00 00 00 00
RIP [<ffffffff81097a2b>] kthread_data+0xb/0x20
RSP <ffffc90006f5fe78>
CR2: ffffffffffffffd8
---[ end trace 41feb073b5b2881c ]---
Fixing recursive fault but reboot is needed!
INFO: rcu_sched detected stalls on CPUs/tasks:
7-...: (2 GPs behind) idle=b43/140000000000000/0 softirq=7775986/7775986 fqs=794
(detected by 4, t=5255 jiffies, g=985507, c=985506, q=549)
Task dump for CPU 7:
kworker/u16:2 D 0 8609 2 0x00000008
ffffc90006f5fc58 ffffffff8107d522 ffffc90006f5fc80 ffffc90006f5fc70
ffffffff812653ba ffffc90006f5fc80 ffffffff812653ea ffffc90006f5fca8
ffffffff812f6842 ffffffff8199e200 000000000000000a ffffffff81968c2d
Call Trace:
[<ffffffff8107d522>] ? oops_exit+0x22/0x30
[<ffffffff812653ba>] ? __delay+0xa/0x10
[<ffffffff812653ea>] ? __const_udelay+0x2a/0x30
[<ffffffff812f6842>] wait_for_xmitr+0x32/0x90
[<ffffffff812653ba>] ? __delay+0xa/0x10
[<ffffffff812653ea>] ? __const_udelay+0x2a/0x30
[<ffffffff812f6842>] ? wait_for_xmitr+0x32/0x90
[<ffffffff812f98f4>] ? serial8250_console_write+0x204/0x250
[<ffffffff8103f0ff>] ? bad_area_nosemaphore+0xf/0x20
[<ffffffff812f4c39>] ? univ8250_console_write+0x19/0x20
[<ffffffff810f258f>] ? irq_work_queue+0xf/0x70
[<ffffffff810ba3ff>] ? wake_up_klogd+0x2f/0x40
[<ffffffff810ba92f>] ? console_unlock+0x51f/0x550
[<ffffffff810babd3>] ? vprintk_emit+0x273/0x430
[<ffffffff814c67cb>] ? _raw_spin_lock+0x1b/0x20
[<ffffffff814c302b>] ? __schedule+0x6b/0x580
[<ffffffff814c3571>] ? schedule+0x31/0x80
[<ffffffff81080e14>] ? do_exit+0x884/0xab0
[<ffffffff814c8417>] ? rewind_stack_do_exit+0x17/0x19
rcu_sched kthread starved for 3604 jiffies! g985507 c985506 f0x0 RCU_GP_WAIT_FQS(3) ->state=0x1
rcu_sched S 0 7 2 0x00000000
0000000000000000 ffff88046182a580 ffff88046d8f4800 ffff88047fc55300
ffff88046b1e3c00 ffffc9000003bdb8 ffffffff814c314e 0000000000000282
ffff88046d8f4800 ffff88047fc4eb80 ffff88047fc4eb80 ffffc9000003bdf0
Call Trace:
[<ffffffff814c314e>] ? __schedule+0x18e/0x580
[<ffffffff814c3571>] schedule+0x31/0x80
[<ffffffff814c5ba9>] schedule_timeout+0xf9/0x180
[<ffffffff810c9ce0>] ? del_timer_sync+0x50/0x50
[<ffffffff810c733e>] rcu_gp_kthread+0x3be/0x7d0
[<ffffffff810c6f80>] ? force_qs_rnp+0x180/0x180
[<ffffffff81097107>] kthread+0xd7/0xf0
[<ffffffff81097030>] ? kthread_park+0x60/0x60
[<ffffffff81097030>] ? kthread_park+0x60/0x60
[<ffffffff814c6c92>] ret_from_fork+0x22/0x30
INFO: rcu_sched detected stalls on CPUs/tasks:
7-...: (2 GPs behind) idle=b43/140000000000000/0 softirq=7775986/7775986 fqs=5824
(detected by 0, t=21007 jiffies, g=985507, c=985506, q=568)
Task dump for CPU 7:
kworker/u16:2 D 0 8609 2 0x00000008
ffffc90006f5fc58 ffffffff8107d522 ffffc90006f5fc80 ffffc90006f5fc70
ffffffff812653ba ffffc90006f5fc80 ffffffff812653ea ffffc90006f5fca8
ffffffff812f6842 ffffffff8199e200 000000000000000a ffffffff81968c2d
Call Trace:
[<ffffffff8107d522>] ? oops_exit+0x22/0x30
[<ffffffff812653ba>] ? __delay+0xa/0x10
[<ffffffff812653ea>] ? __const_udelay+0x2a/0x30
[<ffffffff812f6842>] wait_for_xmitr+0x32/0x90
[<ffffffff812653ba>] ? __delay+0xa/0x10
[<ffffffff812653ea>] ? __const_udelay+0x2a/0x30
[<ffffffff812f6842>] ? wait_for_xmitr+0x32/0x90
[<ffffffff812f98f4>] ? serial8250_console_write+0x204/0x250
[<ffffffff8103f0ff>] ? bad_area_nosemaphore+0xf/0x20
[<ffffffff812f4c39>] ? univ8250_console_write+0x19/0x20
[<ffffffff810f258f>] ? irq_work_queue+0xf/0x70
[<ffffffff810ba3ff>] ? wake_up_klogd+0x2f/0x40
[<ffffffff810ba92f>] ? console_unlock+0x51f/0x550
[<ffffffff810babd3>] ? vprintk_emit+0x273/0x430
[<ffffffff814c67cb>] ? _raw_spin_lock+0x1b/0x20
[<ffffffff814c302b>] ? __schedule+0x6b/0x580
[<ffffffff814c3571>] ? schedule+0x31/0x80
[<ffffffff81080e14>] ? do_exit+0x884/0xab0
[<ffffffff814c8417>] ? rewind_stack_do_exit+0x17/0x19
rcu_sched kthread starved for 5402 jiffies! g985507 c985506 f0x0 RCU_GP_WAIT_FQS(3) ->state=0x1
rcu_sched S 0 7 2 0x00000000
0000000000000000 ffff88046182a580 ffff88046d8f4800 ffff88047fd15300
ffff88046d96a400 ffffc9000003bdb8 ffffffff814c314e 0000000000000282
ffff88046d8f4800 ffff88047fd0eb80 ffff88047fd0eb80 ffffc9000003bdf0
Call Trace:
[<ffffffff814c314e>] ? __schedule+0x18e/0x580
[<ffffffff814c3571>] schedule+0x31/0x80
[<ffffffff814c5ba9>] schedule_timeout+0xf9/0x180
[<ffffffff810c9ce0>] ? del_timer_sync+0x50/0x50
[<ffffffff810c733e>] rcu_gp_kthread+0x3be/0x7d0
[<ffffffff810c6f80>] ? force_qs_rnp+0x180/0x180
[<ffffffff81097107>] kthread+0xd7/0xf0
[<ffffffff81097030>] ? kthread_park+0x60/0x60
[<ffffffff81097030>] ? kthread_park+0x60/0x60
[<ffffffff814c6c92>] ret_from_fork+0x22/0x30
INFO: rcu_sched detected stalls on CPUs/tasks:
7-...: (2 GPs behind) idle=b43/140000000000000/0 softirq=7775986/7775986 fqs=13674
(detected by 0, t=36762 jiffies, g=985507, c=985506, q=570)
Task dump for CPU 7:
kworker/u16:2 D 0 8609 2 0x00000008
ffffc90006f5fc58 ffffffff8107d522 ffffc90006f5fc80 ffffc90006f5fc70
ffffffff812653ba ffffc90006f5fc80 ffffffff812653ea ffffc90006f5fca8
ffffffff812f6842 ffffffff8199e200 000000000000000a ffffffff81968c2d
Call Trace:
[<ffffffff8107d522>] ? oops_exit+0x22/0x30
[<ffffffff812653ba>] ? __delay+0xa/0x10
[<ffffffff812653ea>] ? __const_udelay+0x2a/0x30
[<ffffffff812f6842>] wait_for_xmitr+0x32/0x90
[<ffffffff812653ba>] ? __delay+0xa/0x10
[<ffffffff812653ea>] ? __const_udelay+0x2a/0x30
[<ffffffff812f6842>] ? wait_for_xmitr+0x32/0x90
[<ffffffff812f98f4>] ? serial8250_console_write+0x204/0x250
[<ffffffff8103f0ff>] ? bad_area_nosemaphore+0xf/0x20
[<ffffffff812f4c39>] ? univ8250_console_write+0x19/0x20
[<ffffffff810f258f>] ? irq_work_queue+0xf/0x70
[<ffffffff810ba3ff>] ? wake_up_klogd+0x2f/0x40
[<ffffffff810ba92f>] ? console_unlock+0x51f/0x550
[<ffffffff810babd3>] ? vprintk_emit+0x273/0x430
[<ffffffff814c67cb>] ? _raw_spin_lock+0x1b/0x20
[<ffffffff814c302b>] ? __schedule+0x6b/0x580
[<ffffffff814c3571>] ? schedule+0x31/0x80
[<ffffffff81080e14>] ? do_exit+0x884/0xab0
[<ffffffff814c8417>] ? rewind_stack_do_exit+0x17/0x19
INFO: rcu_sched detected stalls on CPUs/tasks:
7-...: (2 GPs behind) idle=b43/140000000000000/0 softirq=7775986/7775986 fqs=21535
(detected by 0, t=52517 jiffies, g=985507, c=985506, q=574)
Task dump for CPU 7:
kworker/u16:2 D 0 8609 2 0x00000008
ffffc90006f5fc58 ffffffff8107d522 ffffc90006f5fc80 ffffc90006f5fc70
ffffffff812653ba ffffc90006f5fc80 ffffffff812653ea ffffc90006f5fca8
ffffffff812f6842 ffffffff8199e200 000000000000000a ffffffff81968c2d
Call Trace:
[<ffffffff8107d522>] ? oops_exit+0x22/0x30
[<ffffffff812653ba>] ? __delay+0xa/0x10
[<ffffffff812653ea>] ? __const_udelay+0x2a/0x30
[<ffffffff812f6842>] wait_for_xmitr+0x32/0x90
[<ffffffff812653ba>] ? __delay+0xa/0x10
[<ffffffff812653ea>] ? __const_udelay+0x2a/0x30
[<ffffffff812f6842>] ? wait_for_xmitr+0x32/0x90
[<ffffffff812f98f4>] ? serial8250_console_write+0x204/0x250
[<ffffffff8103f0ff>] ? bad_area_nosemaphore+0xf/0x20
[<ffffffff812f4c39>] ? univ8250_console_write+0x19/0x20
[<ffffffff810f258f>] ? irq_work_queue+0xf/0x70
[<ffffffff810ba3ff>] ? wake_up_klogd+0x2f/0x40
[<ffffffff810ba92f>] ? console_unlock+0x51f/0x550
[<ffffffff810babd3>] ? vprintk_emit+0x273/0x430
[<ffffffff814c67cb>] ? _raw_spin_lock+0x1b/0x20
[<ffffffff814c302b>] ? __schedule+0x6b/0x580
[<ffffffff814c3571>] ? schedule+0x31/0x80
[<ffffffff81080e14>] ? do_exit+0x884/0xab0
[<ffffffff814c8417>] ? rewind_stack_do_exit+0x17/0x19
INFO: rcu_sched detected stalls on CPUs/tasks:
7-...: (2 GPs behind) idle=b43/140000000000000/0 softirq=7775986/7775986 fqs=29396
(detected by 0, t=68272 jiffies, g=985507, c=985506, q=576)
Task dump for CPU 7:
kworker/u16:2 D 0 8609 2 0x00000008
ffffc90006f5fc58 ffffffff8107d522 ffffc90006f5fc80 ffffc90006f5fc70
ffffffff812653ba ffffc90006f5fc80 ffffffff812653ea ffffc90006f5fca8
ffffffff812f6842 ffffffff8199e200 000000000000000a ffffffff81968c2d
Call Trace:
[<ffffffff8107d522>] ? oops_exit+0x22/0x30
[<ffffffff812653ba>] ? __delay+0xa/0x10
[<ffffffff812653ea>] ? __const_udelay+0x2a/0x30
[<ffffffff812f6842>] wait_for_xmitr+0x32/0x90
[<ffffffff812653ba>] ? __delay+0xa/0x10
[<ffffffff812653ea>] ? __const_udelay+0x2a/0x30
[<ffffffff812f6842>] ? wait_for_xmitr+0x32/0x90
[<ffffffff812f98f4>] ? serial8250_console_write+0x204/0x250
[<ffffffff8103f0ff>] ? bad_area_nosemaphore+0xf/0x20
[<ffffffff812f4c39>] ? univ8250_console_write+0x19/0x20
[<ffffffff810f258f>] ? irq_work_queue+0xf/0x70
[<ffffffff810ba3ff>] ? wake_up_klogd+0x2f/0x40
[<ffffffff810ba92f>] ? console_unlock+0x51f/0x550
[<ffffffff810babd3>] ? vprintk_emit+0x273/0x430
[<ffffffff814c67cb>] ? _raw_spin_lock+0x1b/0x20
[<ffffffff814c302b>] ? __schedule+0x6b/0x580
[<ffffffff814c3571>] ? schedule+0x31/0x80
[<ffffffff81080e14>] ? do_exit+0x884/0xab0
[<ffffffff814c8417>] ? rewind_stack_do_exit+0x17/0x19
INFO: rcu_sched detected stalls on CPUs/tasks:
7-...: (2 GPs behind) idle=b43/140000000000000/0 softirq=7775986/7775986 fqs=37257
(detected by 0, t=84027 jiffies, g=985507, c=985506, q=580)
Task dump for CPU 7:
kworker/u16:2 D 0 8609 2 0x00000008
ffffc90006f5fc58 ffffffff8107d522 ffffc90006f5fc80 ffffc90006f5fc70
ffffffff812653ba ffffc90006f5fc80 ffffffff812653ea ffffc90006f5fca8
ffffffff812f6842 ffffffff8199e200 000000000000000a ffffffff81968c2d
Call Trace:
[<ffffffff8107d522>] ? oops_exit+0x22/0x30
[<ffffffff812653ba>] ? __delay+0xa/0x10
[<ffffffff812653ea>] ? __const_udelay+0x2a/0x30
[<ffffffff812f6842>] wait_for_xmitr+0x32/0x90
[<ffffffff812653ba>] ? __delay+0xa/0x10
[<ffffffff812653ea>] ? __const_udelay+0x2a/0x30
[<ffffffff812f6842>] ? wait_for_xmitr+0x32/0x90
[<ffffffff812f98f4>] ? serial8250_console_write+0x204/0x250
[<ffffffff8103f0ff>] ? bad_area_nosemaphore+0xf/0x20
[<ffffffff812f4c39>] ? univ8250_console_write+0x19/0x20
[<ffffffff810f258f>] ? irq_work_queue+0xf/0x70
[<ffffffff810ba3ff>] ? wake_up_klogd+0x2f/0x40
[<ffffffff810ba92f>] ? console_unlock+0x51f/0x550
[<ffffffff810babd3>] ? vprintk_emit+0x273/0x430
[<ffffffff814c67cb>] ? _raw_spin_lock+0x1b/0x20
[<ffffffff814c302b>] ? __schedule+0x6b/0x580
[<ffffffff814c3571>] ? schedule+0x31/0x80
[<ffffffff81080e14>] ? do_exit+0x884/0xab0
[<ffffffff814c8417>] ? rewind_stack_do_exit+0x17/0x19
INFO: rcu_sched detected stalls on CPUs/tasks:
7-...: (2 GPs behind) idle=b43/140000000000000/0 softirq=7775986/7775986 fqs=45118
(detected by 0, t=99782 jiffies, g=985507, c=985506, q=582)
Task dump for CPU 7:
kworker/u16:2 D 0 8609 2 0x00000008
ffffc90006f5fc58 ffffffff8107d522 ffffc90006f5fc80 ffffc90006f5fc70
ffffffff812653ba ffffc90006f5fc80 ffffffff812653ea ffffc90006f5fca8
ffffffff812f6842 ffffffff8199e200 000000000000000a ffffffff81968c2d
Call Trace:
[<ffffffff8107d522>] ? oops_exit+0x22/0x30
[<ffffffff812653ba>] ? __delay+0xa/0x10
[<ffffffff812653ea>] ? __const_udelay+0x2a/0x30
[<ffffffff812f6842>] wait_for_xmitr+0x32/0x90
[<ffffffff812653ba>] ? __delay+0xa/0x10
[<ffffffff812653ea>] ? __const_udelay+0x2a/0x30
[<ffffffff812f6842>] ? wait_for_xmitr+0x32/0x90
[<ffffffff812f98f4>] ? serial8250_console_write+0x204/0x250
[<ffffffff8103f0ff>] ? bad_area_nosemaphore+0xf/0x20
[<ffffffff812f4c39>] ? univ8250_console_write+0x19/0x20
[<ffffffff810f258f>] ? irq_work_queue+0xf/0x70
[<ffffffff810ba3ff>] ? wake_up_klogd+0x2f/0x40
[<ffffffff810ba92f>] ? console_unlock+0x51f/0x550
[<ffffffff810babd3>] ? vprintk_emit+0x273/0x430
[<ffffffff814c67cb>] ? _raw_spin_lock+0x1b/0x20
[<ffffffff814c302b>] ? __schedule+0x6b/0x580
[<ffffffff814c3571>] ? schedule+0x31/0x80
[<ffffffff81080e14>] ? do_exit+0x884/0xab0
[<ffffffff814c8417>] ? rewind_stack_do_exit+0x17/0x19
INFO: rcu_sched detected stalls on CPUs/tasks:
7-...: (2 GPs behind) idle=b43/140000000000000/0 softirq=7775986/7775986 fqs=52979
(detected by 0, t=115537 jiffies, g=985507, c=985506, q=586)
Task dump for CPU 7:
kworker/u16:2 D 0 8609 2 0x00000008
ffffc90006f5fc58 ffffffff8107d522 ffffc90006f5fc80 ffffc90006f5fc70
ffffffff812653ba ffffc90006f5fc80 ffffffff812653ea ffffc90006f5fca8
ffffffff812f6842 ffffffff8199e200 000000000000000a ffffffff81968c2d
Call Trace:
[<ffffffff8107d522>] ? oops_exit+0x22/0x30
[<ffffffff812653ba>] ? __delay+0xa/0x10
[<ffffffff812653ea>] ? __const_udelay+0x2a/0x30
[<ffffffff812f6842>] wait_for_xmitr+0x32/0x90
[<ffffffff812653ba>] ? __delay+0xa/0x10
[<ffffffff812653ea>] ? __const_udelay+0x2a/0x30
[<ffffffff812f6842>] ? wait_for_xmitr+0x32/0x90
[<ffffffff812f98f4>] ? serial8250_console_write+0x204/0x250
[<ffffffff8103f0ff>] ? bad_area_nosemaphore+0xf/0x20
[<ffffffff812f4c39>] ? univ8250_console_write+0x19/0x20
[<ffffffff810f258f>] ? irq_work_queue+0xf/0x70
[<ffffffff810ba3ff>] ? wake_up_klogd+0x2f/0x40
[<ffffffff810ba92f>] ? console_unlock+0x51f/0x550
[<ffffffff810babd3>] ? vprintk_emit+0x273/0x430
[<ffffffff814c67cb>] ? _raw_spin_lock+0x1b/0x20
[<ffffffff814c302b>] ? __schedule+0x6b/0x580
[<ffffffff814c3571>] ? schedule+0x31/0x80
[<ffffffff81080e14>] ? do_exit+0x884/0xab0
[<ffffffff814c8417>] ? rewind_stack_do_exit+0x17/0x19
INFO: rcu_sched detected stalls on CPUs/tasks:
7-...: (2 GPs behind) idle=b43/140000000000000/0 softirq=7775986/7775986 fqs=60840
(detected by 0, t=131292 jiffies, g=985507, c=985506, q=588)
Task dump for CPU 7:
kworker/u16:2 D 0 8609 2 0x00000008
ffffc90006f5fc58 ffffffff8107d522 ffffc90006f5fc80 ffffc90006f5fc70
ffffffff812653ba ffffc90006f5fc80 ffffffff812653ea ffffc90006f5fca8
ffffffff812f6842 ffffffff8199e200 000000000000000a ffffffff81968c2d
Call Trace:
[<ffffffff8107d522>] ? oops_exit+0x22/0x30
[<ffffffff812653ba>] ? __delay+0xa/0x10
[<ffffffff812653ea>] ? __const_udelay+0x2a/0x30
[<ffffffff812f6842>] wait_for_xmitr+0x32/0x90
[<ffffffff812653ba>] ? __delay+0xa/0x10
[<ffffffff812653ea>] ? __const_udelay+0x2a/0x30
[<ffffffff812f6842>] ? wait_for_xmitr+0x32/0x90
[<ffffffff812f98f4>] ? serial8250_console_write+0x204/0x250
[<ffffffff8103f0ff>] ? bad_area_nosemaphore+0xf/0x20
[<ffffffff812f4c39>] ? univ8250_console_write+0x19/0x20
[<ffffffff810f258f>] ? irq_work_queue+0xf/0x70
[<ffffffff810ba3ff>] ? wake_up_klogd+0x2f/0x40
[<ffffffff810ba92f>] ? console_unlock+0x51f/0x550
[<ffffffff810babd3>] ? vprintk_emit+0x273/0x430
[<ffffffff814c67cb>] ? _raw_spin_lock+0x1b/0x20
[<ffffffff814c302b>] ? __schedule+0x6b/0x580
[<ffffffff814c3571>] ? schedule+0x31/0x80
[<ffffffff81080e14>] ? do_exit+0x884/0xab0
[<ffffffff814c8417>] ? rewind_stack_do_exit+0x17/0x19
INFO: rcu_sched detected stalls on CPUs/tasks:
7-...: (2 GPs behind) idle=b43/140000000000000/0 softirq=7775986/7775986 fqs=68701
(detected by 0, t=147047 jiffies, g=985507, c=985506, q=588)
Task dump for CPU 7:
kworker/u16:2 D 0 8609 2 0x00000008
ffffc90006f5fc58 ffffffff8107d522 ffffc90006f5fc80 ffffc90006f5fc70
ffffffff812653ba ffffc90006f5fc80 ffffffff812653ea ffffc90006f5fca8
ffffffff812f6842 ffffffff8199e200 000000000000000a ffffffff81968c2d
Call Trace:
[<ffffffff8107d522>] ? oops_exit+0x22/0x30
[<ffffffff812653ba>] ? __delay+0xa/0x10
[<ffffffff812653ea>] ? __const_udelay+0x2a/0x30
[<ffffffff812f6842>] wait_for_xmitr+0x32/0x90
[<ffffffff812653ba>] ? __delay+0xa/0x10
[<ffffffff812653ea>] ? __const_udelay+0x2a/0x30
[<ffffffff812f6842>] ? wait_for_xmitr+0x32/0x90
[<ffffffff812f98f4>] ? serial8250_console_write+0x204/0x250
[<ffffffff8103f0ff>] ? bad_area_nosemaphore+0xf/0x20
[<ffffffff812f4c39>] ? univ8250_console_write+0x19/0x20
[<ffffffff810f258f>] ? irq_work_queue+0xf/0x70
[<ffffffff810ba3ff>] ? wake_up_klogd+0x2f/0x40
[<ffffffff810ba92f>] ? console_unlock+0x51f/0x550
[<ffffffff810babd3>] ? vprintk_emit+0x273/0x430
[<ffffffff814c67cb>] ? _raw_spin_lock+0x1b/0x20
[<ffffffff814c302b>] ? __schedule+0x6b/0x580
[<ffffffff814c3571>] ? schedule+0x31/0x80
[<ffffffff81080e14>] ? do_exit+0x884/0xab0
[<ffffffff814c8417>] ? rewind_stack_do_exit+0x17/0x19
INFO: rcu_sched detected stalls on CPUs/tasks:
7-...: (2 GPs behind) idle=b43/140000000000000/0 softirq=7775986/7775986 fqs=76562
(detected by 0, t=162802 jiffies, g=985507, c=985506, q=588)
Task dump for CPU 7:
kworker/u16:2 D 0 8609 2 0x00000008
ffffc90006f5fc58 ffffffff8107d522 ffffc90006f5fc80 ffffc90006f5fc70
ffffffff812653ba ffffc90006f5fc80 ffffffff812653ea ffffc90006f5fca8
ffffffff812f6842 ffffffff8199e200 000000000000000a ffffffff81968c2d
Call Trace:
[<ffffffff8107d522>] ? oops_exit+0x22/0x30
[<ffffffff812653ba>] ? __delay+0xa/0x10
[<ffffffff812653ea>] ? __const_udelay+0x2a/0x30
[<ffffffff812f6842>] wait_for_xmitr+0x32/0x90
[<ffffffff812653ba>] ? __delay+0xa/0x10
[<ffffffff812653ea>] ? __const_udelay+0x2a/0x30
[<ffffffff812f6842>] ? wait_for_xmitr+0x32/0x90
[<ffffffff812f98f4>] ? serial8250_console_write+0x204/0x250
[<ffffffff8103f0ff>] ? bad_area_nosemaphore+0xf/0x20
[<ffffffff812f4c39>] ? univ8250_console_write+0x19/0x20
[<ffffffff810f258f>] ? irq_work_queue+0xf/0x70
[<ffffffff810ba3ff>] ? wake_up_klogd+0x2f/0x40
[<ffffffff810ba92f>] ? console_unlock+0x51f/0x550
[<ffffffff810babd3>] ? vprintk_emit+0x273/0x430
[<ffffffff814c67cb>] ? _raw_spin_lock+0x1b/0x20
[<ffffffff814c302b>] ? __schedule+0x6b/0x580
[<ffffffff814c3571>] ? schedule+0x31/0x80
[<ffffffff81080e14>] ? do_exit+0x884/0xab0
[<ffffffff814c8417>] ? rewind_stack_do_exit+0x17/0x19
INFO: rcu_sched detected stalls on CPUs/tasks:
7-...: (2 GPs behind) idle=b43/140000000000000/0 softirq=7775986/7775986 fqs=84423
(detected by 0, t=178557 jiffies, g=985507, c=985506, q=588)
Task dump for CPU 7:
kworker/u16:2 D 0 8609 2 0x00000008
ffffc90006f5fc58 ffffffff8107d522 ffffc90006f5fc80 ffffc90006f5fc70
ffffffff812653ba ffffc90006f5fc80 ffffffff812653ea ffffc90006f5fca8
ffffffff812f6842 ffffffff8199e200 000000000000000a ffffffff81968c2d
Call Trace:
[<ffffffff8107d522>] ? oops_exit+0x22/0x30
[<ffffffff812653ba>] ? __delay+0xa/0x10
[<ffffffff812653ea>] ? __const_udelay+0x2a/0x30
[<ffffffff812f6842>] wait_for_xmitr+0x32/0x90
[<ffffffff812653ba>] ? __delay+0xa/0x10
[<ffffffff812653ea>] ? __const_udelay+0x2a/0x30
[<ffffffff812f6842>] ? wait_for_xmitr+0x32/0x90
[<ffffffff812f98f4>] ? serial8250_console_write+0x204/0x250
[<ffffffff8103f0ff>] ? bad_area_nosemaphore+0xf/0x20
[<ffffffff812f4c39>] ? univ8250_console_write+0x19/0x20
[<ffffffff810f258f>] ? irq_work_queue+0xf/0x70
[<ffffffff810ba3ff>] ? wake_up_klogd+0x2f/0x40
[<ffffffff810ba92f>] ? console_unlock+0x51f/0x550
[<ffffffff810babd3>] ? vprintk_emit+0x273/0x430
[<ffffffff814c67cb>] ? _raw_spin_lock+0x1b/0x20
[<ffffffff814c302b>] ? __schedule+0x6b/0x580
[<ffffffff814c3571>] ? schedule+0x31/0x80
[<ffffffff81080e14>] ? do_exit+0x884/0xab0
[<ffffffff814c8417>] ? rewind_stack_do_exit+0x17/0x19
INFO: rcu_sched detected stalls on CPUs/tasks:
7-...: (2 GPs behind) idle=b43/140000000000000/0 softirq=7775986/7775986 fqs=92284
(detected by 0, t=194312 jiffies, g=985507, c=985506, q=588)
Task dump for CPU 7:
kworker/u16:2 D 0 8609 2 0x00000008
ffffc90006f5fc58 ffffffff8107d522 ffffc90006f5fc80 ffffc90006f5fc70
ffffffff812653ba ffffc90006f5fc80 ffffffff812653ea ffffc90006f5fca8
ffffffff812f6842 ffffffff8199e200 000000000000000a ffffffff81968c2d
Call Trace:
[<ffffffff8107d522>] ? oops_exit+0x22/0x30
[<ffffffff812653ba>] ? __delay+0xa/0x10
[<ffffffff812653ea>] ? __const_udelay+0x2a/0x30
[<ffffffff812f6842>] wait_for_xmitr+0x32/0x90
[<ffffffff812653ba>] ? __delay+0xa/0x10
[<ffffffff812653ea>] ? __const_udelay+0x2a/0x30
[<ffffffff812f6842>] ? wait_for_xmitr+0x32/0x90
[<ffffffff812f98f4>] ? serial8250_console_write+0x204/0x250
[<ffffffff8103f0ff>] ? bad_area_nosemaphore+0xf/0x20
[<ffffffff812f4c39>] ? univ8250_console_write+0x19/0x20
[<ffffffff810f258f>] ? irq_work_queue+0xf/0x70
[<ffffffff810ba3ff>] ? wake_up_klogd+0x2f/0x40
[<ffffffff810ba92f>] ? console_unlock+0x51f/0x550
[<ffffffff810babd3>] ? vprintk_emit+0x273/0x430
[<ffffffff814c67cb>] ? _raw_spin_lock+0x1b/0x20
[<ffffffff814c302b>] ? __schedule+0x6b/0x580
[<ffffffff814c3571>] ? schedule+0x31/0x80
[<ffffffff81080e14>] ? do_exit+0x884/0xab0
[<ffffffff814c8417>] ? rewind_stack_do_exit+0x17/0x19
INFO: rcu_sched detected stalls on CPUs/tasks:
7-...: (2 GPs behind) idle=b43/140000000000000/0 softirq=7775986/7775986 fqs=100145
(detected by 0, t=210067 jiffies, g=985507, c=985506, q=588)
Task dump for CPU 7:
kworker/u16:2 D 0 8609 2 0x00000008
ffffc90006f5fc58 ffffffff8107d522 ffffc90006f5fc80 ffffc90006f5fc70
ffffffff812653ba ffffc90006f5fc80 ffffffff812653ea ffffc90006f5fca8
ffffffff812f6842 ffffffff8199e200 000000000000000a ffffffff81968c2d
Call Trace:
[<ffffffff8107d522>] ? oops_exit+0x22/0x30
[<ffffffff812653ba>] ? __delay+0xa/0x10
[<ffffffff812653ea>] ? __const_udelay+0x2a/0x30
[<ffffffff812f6842>] wait_for_xmitr+0x32/0x90
[<ffffffff812653ba>] ? __delay+0xa/0x10
[<ffffffff812653ea>] ? __const_udelay+0x2a/0x30
[<ffffffff812f6842>] ? wait_for_xmitr+0x32/0x90
[<ffffffff812f98f4>] ? serial8250_console_write+0x204/0x250
[<ffffffff8103f0ff>] ? bad_area_nosemaphore+0xf/0x20
[<ffffffff812f4c39>] ? univ8250_console_write+0x19/0x20
[<ffffffff810f258f>] ? irq_work_queue+0xf/0x70
[<ffffffff810ba3ff>] ? wake_up_klogd+0x2f/0x40
[<ffffffff810ba92f>] ? console_unlock+0x51f/0x550
[<ffffffff810babd3>] ? vprintk_emit+0x273/0x430
[<ffffffff814c67cb>] ? _raw_spin_lock+0x1b/0x20
[<ffffffff814c302b>] ? __schedule+0x6b/0x580
[<ffffffff814c3571>] ? schedule+0x31/0x80
[<ffffffff81080e14>] ? do_exit+0x884/0xab0
[<ffffffff814c8417>] ? rewind_stack_do_exit+0x17/0x19
INFO: rcu_sched detected stalls on CPUs/tasks:
7-...: (2 GPs behind) idle=b43/140000000000000/0 softirq=7775986/7775986 fqs=108006
(detected by 0, t=225822 jiffies, g=985507, c=985506, q=588)
Task dump for CPU 7:
kworker/u16:2 D 0 8609 2 0x00000008
ffffc90006f5fc58 ffffffff8107d522 ffffc90006f5fc80 ffffc90006f5fc70
ffffffff812653ba ffffc90006f5fc80 ffffffff812653ea ffffc90006f5fca8
ffffffff812f6842 ffffffff8199e200 000000000000000a ffffffff81968c2d
Call Trace:
[<ffffffff8107d522>] ? oops_exit+0x22/0x30
[<ffffffff812653ba>] ? __delay+0xa/0x10
[<ffffffff812653ea>] ? __const_udelay+0x2a/0x30
[<ffffffff812f6842>] wait_for_xmitr+0x32/0x90
[<ffffffff812653ba>] ? __delay+0xa/0x10
[<ffffffff812653ea>] ? __const_udelay+0x2a/0x30
[<ffffffff812f6842>] ? wait_for_xmitr+0x32/0x90
[<ffffffff812f98f4>] ? serial8250_console_write+0x204/0x250
[<ffffffff8103f0ff>] ? bad_area_nosemaphore+0xf/0x20
[<ffffffff812f4c39>] ? univ8250_console_write+0x19/0x20
[<ffffffff810f258f>] ? irq_work_queue+0xf/0x70
[<ffffffff810ba3ff>] ? wake_up_klogd+0x2f/0x40
[<ffffffff810ba92f>] ? console_unlock+0x51f/0x550
[<ffffffff810babd3>] ? vprintk_emit+0x273/0x430
[<ffffffff814c67cb>] ? _raw_spin_lock+0x1b/0x20
[<ffffffff814c302b>] ? __schedule+0x6b/0x580
[<ffffffff814c3571>] ? schedule+0x31/0x80
[<ffffffff81080e14>] ? do_exit+0x884/0xab0
[<ffffffff814c8417>] ? rewind_stack_do_exit+0x17/0x19
INFO: rcu_sched detected stalls on CPUs/tasks:
7-...: (2 GPs behind) idle=b43/140000000000000/0 softirq=7775986/7775986 fqs=115867
(detected by 0, t=241577 jiffies, g=985507, c=985506, q=588)
Task dump for CPU 7:
kworker/u16:2 D 0 8609 2 0x00000008
ffffc90006f5fc58 ffffffff8107d522 ffffc90006f5fc80 ffffc90006f5fc70
ffffffff812653ba ffffc90006f5fc80 ffffffff812653ea ffffc90006f5fca8
ffffffff812f6842 ffffffff8199e200 000000000000000a ffffffff81968c2d
Call Trace:
[<ffffffff8107d522>] ? oops_exit+0x22/0x30
[<ffffffff812653ba>] ? __delay+0xa/0x10
[<ffffffff812653ea>] ? __const_udelay+0x2a/0x30
[<ffffffff812f6842>] wait_for_xmitr+0x32/0x90
[<ffffffff812653ba>] ? __delay+0xa/0x10
[<ffffffff812653ea>] ? __const_udelay+0x2a/0x30
[<ffffffff812f6842>] ? wait_for_xmitr+0x32/0x90
[<ffffffff812f98f4>] ? serial8250_console_write+0x204/0x250
[<ffffffff8103f0ff>] ? bad_area_nosemaphore+0xf/0x20
[<ffffffff812f4c39>] ? univ8250_console_write+0x19/0x20
[<ffffffff810f258f>] ? irq_work_queue+0xf/0x70
[<ffffffff810ba3ff>] ? wake_up_klogd+0x2f/0x40
[<ffffffff810ba92f>] ? console_unlock+0x51f/0x550
[<ffffffff810babd3>] ? vprintk_emit+0x273/0x430
[<ffffffff814c67cb>] ? _raw_spin_lock+0x1b/0x20
[<ffffffff814c302b>] ? __schedule+0x6b/0x580
[<ffffffff814c3571>] ? schedule+0x31/0x80
[<ffffffff81080e14>] ? do_exit+0x884/0xab0
[<ffffffff814c8417>] ? rewind_stack_do_exit+0x17/0x19
INFO: rcu_sched detected stalls on CPUs/tasks:
7-...: (2 GPs behind) idle=b43/140000000000000/0 softirq=7775986/7775986 fqs=123728
(detected by 0, t=257332 jiffies, g=985507, c=985506, q=588)
Task dump for CPU 7:
kworker/u16:2 D 0 8609 2 0x00000008
ffffc90006f5fc58 ffffffff8107d522 ffffc90006f5fc80 ffffc90006f5fc70
ffffffff812653ba ffffc90006f5fc80 ffffffff812653ea ffffc90006f5fca8
ffffffff812f6842 ffffffff8199e200 000000000000000a ffffffff81968c2d
Call Trace:
[<ffffffff8107d522>] ? oops_exit+0x22/0x30
[<ffffffff812653ba>] ? __delay+0xa/0x10
[<ffffffff812653ea>] ? __const_udelay+0x2a/0x30
[<ffffffff812f6842>] wait_for_xmitr+0x32/0x90
[<ffffffff812653ba>] ? __delay+0xa/0x10
[<ffffffff812653ea>] ? __const_udelay+0x2a/0x30
[<ffffffff812f6842>] ? wait_for_xmitr+0x32/0x90
[<ffffffff812f98f4>] ? serial8250_console_write+0x204/0x250
[<ffffffff8103f0ff>] ? bad_area_nosemaphore+0xf/0x20
[<ffffffff812f4c39>] ? univ8250_console_write+0x19/0x20
[<ffffffff810f258f>] ? irq_work_queue+0xf/0x70
[<ffffffff810ba3ff>] ? wake_up_klogd+0x2f/0x40
[<ffffffff810ba92f>] ? console_unlock+0x51f/0x550
[<ffffffff810babd3>] ? vprintk_emit+0x273/0x430
[<ffffffff814c67cb>] ? _raw_spin_lock+0x1b/0x20
[<ffffffff814c302b>] ? __schedule+0x6b/0x580
[<ffffffff814c3571>] ? schedule+0x31/0x80
[<ffffffff81080e14>] ? do_exit+0x884/0xab0
[<ffffffff814c8417>] ? rewind_stack_do_exit+0x17/0x19
INFO: rcu_sched detected stalls on CPUs/tasks:
7-...: (2 GPs behind) idle=b43/140000000000000/0 softirq=7775986/7775986 fqs=131589
(detected by 0, t=273087 jiffies, g=985507, c=985506, q=588)
Task dump for CPU 7:
kworker/u16:2 D 0 8609 2 0x00000008
ffffc90006f5fc58 ffffffff8107d522 ffffc90006f5fc80 ffffc90006f5fc70
ffffffff812653ba ffffc90006f5fc80 ffffffff812653ea ffffc90006f5fca8
ffffffff812f6842 ffffffff8199e200 000000000000000a ffffffff81968c2d
Call Trace:
[<ffffffff8107d522>] ? oops_exit+0x22/0x30
[<ffffffff812653ba>] ? __delay+0xa/0x10
[<ffffffff812653ea>] ? __const_udelay+0x2a/0x30
[<ffffffff812f6842>] wait_for_xmitr+0x32/0x90
[<ffffffff812653ba>] ? __delay+0xa/0x10
[<ffffffff812653ea>] ? __const_udelay+0x2a/0x30
[<ffffffff812f6842>] ? wait_for_xmitr+0x32/0x90
[<ffffffff812f98f4>] ? serial8250_console_write+0x204/0x250
[<ffffffff8103f0ff>] ? bad_area_nosemaphore+0xf/0x20
[<ffffffff812f4c39>] ? univ8250_console_write+0x19/0x20
[<ffffffff810f258f>] ? irq_work_queue+0xf/0x70
[<ffffffff810ba3ff>] ? wake_up_klogd+0x2f/0x40
[<ffffffff810ba92f>] ? console_unlock+0x51f/0x550
[<ffffffff810babd3>] ? vprintk_emit+0x273/0x430
[<ffffffff814c67cb>] ? _raw_spin_lock+0x1b/0x20
[<ffffffff814c302b>] ? __schedule+0x6b/0x580
[<ffffffff814c3571>] ? schedule+0x31/0x80
[<ffffffff81080e14>] ? do_exit+0x884/0xab0
[<ffffffff814c8417>] ? rewind_stack_do_exit+0x17/0x19

index out of range error, after goes install

This has been easy to repro.

sudo /usr/bin/goes stop
sudo ./goes-platina-mk1 install
wait 20 secs
vnet process is dead
index out of range in syslog

Version:

donn@invader8:~/workspace/go/src/github.com/platinasystems/go$ /usr/bin/goes version
0049b61855f30892d00d8a85818cf16212c1f6c3

commit 0049b61855f30892d00d8a85818cf16212c1f6c3
Author: Mark Pleso <[email protected]>
Date:   Tue Feb 28 14:18:49 2017 -0800

    Fixed i2cd mutex

My test:

donn@invader8:~/workspace/go/src/github.com/platinasystems/go$ sudo /usr/bin/goes
 stop
donn@invader8:~/workspace/go/src/github.com/platinasystems/go$

donn@invader8:~/workspace/go/src/github.com/platinasystems/go$ sudo ./goes-platina-mk1 install
donn@invader8:~/workspace/go/src/github.com/platinasystems/go$ ps -e f; date
  PID TTY      STAT   TIME COMMAND
<...>
29491 ?        Ssl    0:00 goes-daemons
29500 ?        Sl     0:00  \_ redisd
29531 ?        Sl     0:00  \_ i2cd
29532 ?        Sl     0:00  \_ telnetd
29537 ?        Sl     0:00  \_ vnetd
29542 ?        Sl     0:00  \_ qsfpio
29548 ?        Sl     0:00  \_ nld
29571 ?        Sl     0:00  \_ uptimed
Wed Mar  1 17:33:20 PST 2017

Wait 20s.

vnet is missing:

donn@invader8:~/workspace/go/src/github.com/platinasystems/go$ ps -e f; date
  PID TTY      STAT   TIME COMMAND
<...>
29491 ?        Ssl    0:00 goes-daemons
29500 ?        Sl     0:00  \_ redisd
29531 ?        Sl     0:00  \_ i2cd
29532 ?        Sl     0:00  \_ telnetd
29542 ?        Sl     0:00  \_ qsfpio
29548 ?        Sl     0:00  \_ nld
29571 ?        Sl     0:00  \_ uptimed
Wed Mar  1 17:33:42 PST 2017

syslog:

Mar  1 17:33:32 invader8 goes.vnetd[29537]: runtime error: index out of range: go
routine 30 [running]:
Mar  1 17:33:32 invader8 goes.vnetd[29537]: runtime/debug.Stack(0xc420043840, 0xa
73780, 0xc42000c0a0)
Mar  1 17:33:32 invader8 goes.vnetd[29537]:         /usr/local/go/src/runtime/deb
ug/stack.go:24 +0x79
Mar  1 17:33:32 invader8 goes.vnetd[29537]: github.com/platinasystems/go/elib/loo
p.(*Loop).doEvent.func1(0xc420186000)
Mar  1 17:33:32 invader8 goes.vnetd[29537]:         /home/donn/workspace/go/src/g
ithub.com/platinasystems/go/elib/loop/event.go:128 +0x91
Mar  1 17:33:32 invader8 goes.vnetd[29537]: panic(0xa73780, 0xc42000c0a0)
Mar  1 17:33:32 invader8 goes.vnetd[29537]:         /usr/local/go/src/runtime/pan
ic.go:458 +0x243
Mar  1 17:33:32 invader8 goes.vnetd[29537]: github.com/platinasystems/go/vnet/dev
ices/ethernet/switch/fe1/internal/fe1a.(*fe1a).ip4_fib_add_del(0xc42021c000, 0xc4
00000000, 0xc42127f8b0, 0x2)
Mar  1 17:33:32 invader8 goes.vnetd[29537]:         /home/donn/workspace/go/src/g
ithub.com/platinasystems/go/vnet/devices/ethernet/switch/fe1/internal/fe1a/fib.go
:286 +0xa61
Mar  1 17:33:32 invader8 goes.vnetd[29537]: github.com/platinasystems/go/vnet/dev
ices/ethernet/switch/fe1/internal/fe1a.(*fe1a).(github.com/platinasystems/go/vnet
/devices/ethernet/switch/fe1/internal/fe1a.ip4_fib_add_del)-fm(0xc400000000, 0xc4
2127f8b0, 0x2)
Mar  1 17:33:32 invader8 goes.vnetd[29537]:         /home/donn/workspace/go/src/g
ithub.com/platinasystems/go/vnet/devices/ethernet/switch/fe1/internal/fe1a/fib.go
:49 +0x56
Mar  1 17:33:32 invader8 goes.vnetd[29537]: github.com/platinasystems/go/vnet/ip4
.(*Fib).addDel(0xc42020c000, 0xc420182700, 0xc42127f8b0, 0x100000000000002, 0x0)
Mar  1 17:33:32 invader8 goes.vnetd[29537]:         /home/donn/workspace/go/src/github.com/platinasystems/go/vnet/ip4/fib.go:217 +0x2cc
Mar  1 17:33:32 invader8 goes.vnetd[29537]: github.com/platinasystems/go/vnet/ip4.(*Main).addDelRoute(0xc420182700, 0xc421280920, 0x200000000, 0x0, 0xaa5ae0, 0x1, 0xc421280920)
Mar  1 17:33:32 invader8 goes.vnetd[29537]:         /home/donn/workspace/go/src/github.com/platinasystems/go/vnet/ip4/fib.go:367 +0xf5
Mar  1 17:33:32 invader8 goes.vnetd[29537]: github.com/platinasystems/go/vnet/ip4.(*Main).(github.com/platinasystems/go/vnet/ip4.addDelRoute)-fm(0xc421280920, 0x200000000, 0x0, 0xc420c061e8, 0x116dc01, 0xad6960)
Mar  1 17:33:32 invader8 goes.vnetd[29537]:         /home/donn/workspace/go/src/github.com/platinasystems/go/vnet/ip4/package.go:43 +0x4d
Mar  1 17:33:32 invader8 goes.vnetd[29537]: github.com/platinasystems/go/vnet/unix.(*Main).ip4IfaddrMsg(0xc420166d80, 0xc42015e2c0, 0x0, 0x0)
Mar  1 17:33:32 invader8 goes.vnetd[29537]:         /home/donn/workspace/go/src/github.com/platinasystems/go/vnet/unix/netlink.go:386 +0x1eb
Mar  1 17:33:32 invader8 goes.vnetd[29537]: github.com/platinasystems/go/vnet/unix.(*netlinkEvent).EventAction(0xc4200f0000)
Mar  1 17:33:32 invader8 goes.vnetd[29537]:         /home/donn/workspace/go/src/github.com/platinasystems/go/vnet/unix/netlink.go:288 +0x3bb
Mar  1 17:33:32 invader8 goes.vnetd[29537]: github.com/platinasystems/go/elib/loop.(*loopEvent).do(0xc4208c1440)
Mar  1 17:33:32 invader8 goes.vnetd[29537]:         /home/donn/workspace/go/src/github.com/platinasystems/go/elib/loop/event.go:118 +0x34
Mar  1 17:33:32 invader8 goes.vnetd[29537]: github.com/platinasystems/go/elib/loop.(*Loop).doEvent(0xc420186000, 0xc4208c1440)
Mar  1 17:33:32 invader8 goes.vnetd[29537]:         /home/donn/workspace/go/src/github.com/platinasystems/go/elib/loop/event.go:133 +0x51
Mar  1 17:33:32 invader8 goes.vnetd[29537]: github.com/platinasystems/go/elib/loop.(*Loop).eventHandler(0xc420186000, 0x127e760, 0xc420186548)
Mar  1 17:33:32 invader8 goes.vnetd[29537]:         /home/donn/workspace/go/src/github.com/platinasystems/go/elib/loop/event.go:140 +0x86
Mar  1 17:33:32 invader8 goes.vnetd[29537]: created by github.com/platinasystems/go/elib/loop.(*Loop).startHandler
Mar  1 17:33:32 invader8 goes.vnetd[29537]:         /home/donn/workspace/go/src/github.com/platinasystems/go/elib/loop/event.go:153 +0x124
Mar  1 17:33:41 invader8 goes.vnetd[29537]: done

Counter constipation

After many packets (60billion?) - redis counter update from vnet becomes slow and inconsistent.

autoneg ports sometimes do not come up after goes start

Platina to platina ports configured for autoneg sometimes doesn't come up after goes start.

The port recovers only after issuing another "speed autoneg" setting; or with physical port tx enable disable (with debug code); or setting the autoneg restart bit (with debug code). i.e. anything that triggers another autoneg attempt.

mmu port flush timeout

New code was adding this morning that flushes the mmu for Issue #1. Sometimes it will crash when admin up/down an interface. Eliot & Don are investigating.

Apr 19 15:07:11 invader24 kernel: ixgbe 0000:03:00.0 eth1: VF Reset msg received from vf 0
Apr 19 15:07:12 invader24 goes.vnetd[8382]: mmu port flush timeout true: goroutine 27 [running]:
Apr 19 15:07:12 invader24 goes.vnetd[8382]: runtime/debug.Stack(0x0, 0x0, 0x0)
Apr 19 15:07:12 invader24 goes.vnetd[8382]: /usr/local/go/src/runtime/debug/stack.go:24 +0x80
Apr 19 15:07:12 invader24 goes.vnetd[8382]: github.com/platinasystems/go/elib/loop.(*Loop).doEvent.func1(0xc820192000)
Apr 19 15:07:12 invader24 goes.vnetd[8382]: /home/stig/go/src/github.com/platinasystems/go/elib/loop/event.go:128 +0xa2
Apr 19 15:07:12 invader24 goes.vnetd[8382]: panic(0xd9b0a0, 0xc8211b4950)
Apr 19 15:07:12 invader24 goes.vnetd[8382]: /usr/local/go/src/runtime/panic.go:426 +0x4e9
Apr 19 15:07:12 invader24 goes.vnetd[8382]: github.com/platinasystems/go/vnet/devices/ethernet/switch/fe1/internal/fe1a.(*Port).mmu_port_flush(0xc82036b628, 0xc82021e000)
Apr 19 15:07:12 invader24 goes.vnetd[8382]: /home/stig/go/src/github.com/platinasystems/go/vnet/devices/ethernet/switch/fe1/internal/fe1a/mmu.go:391 +0x27f
Apr 19 15:07:12 invader24 goes.vnetd[8382]: github.com/platinasystems/go/vnet/devices/ethernet/switch/fe1/internal/fe1a.(*fe1a).swIfAdminUpDown(0xc82021e000, 0xc820192000, 0x44, 0x0, 0x0)
Apr 19 15:07:12 invader24 goes.vnetd[8382]: /home/stig/go/src/github.com/platinasystems/go/vnet/devices/ethernet/switch/fe1/internal/fe1a/l3.go:62 +0x2ff
Apr 19 15:07:12 invader24 goes.vnetd[8382]: github.com/platinasystems/go/vnet/devices/ethernet/switch/fe1/internal/fe1a.(*fe1a).(github.com/platinasystems/go/vnet/devices/ethernet/switch/fe1/internal/fe1a.swIfAdminUpDown)-fm(0xc820192000, 0x44, 0x0, 0x0)
Apr 19 15:07:12 invader24 goes.vnetd[8382]: /home/stig/go/src/github.com/platinasystems/go/vnet/devices/ethernet/switch/fe1/internal/fe1a/l3.go:26 +0x49
Apr 19 15:07:12 invader24 goes.vnetd[8382]: github.com/platinasystems/go/vnet.(*swIf).SetAdminUp(0xc820666840, 0xc820192000, 0xc820178100, 0x0, 0x0)
Apr 19 15:07:12 invader24 goes.vnetd[8382]: /home/stig/go/src/github.com/platinasystems/go/vnet/interface.go:225 +0xce
Apr 19 15:07:12 invader24 goes.vnetd[8382]: github.com/platinasystems/go/vnet.Si.SetAdminUp(0xc800000044, 0xc820192000, 0xffffff00, 0x0, 0x0)
Apr 19 15:07:12 invader24 goes.vnetd[8382]: /home/stig/go/src/github.com/platinasystems/go/vnet/interface.go:235 +0x8a
Apr 19 15:07:12 invader24 goes.vnetd[8382]: github.com/platinasystems/go/vnet/unix.(*netlinkEvent).EventAction(0xc8208ee0c0)
Apr 19 15:07:12 invader24 goes.vnetd[8382]: /home/stig/go/src/github.com/platinasystems/go/vnet/unix/netlink.go:330 +0x8fa
Apr 19 15:07:12 invader24 goes.vnetd[8382]: github.com/platinasystems/go/elib/loop.(*loopEvent).do(0xc8208ee120)
Apr 19 15:07:12 invader24 goes.vnetd[8382]: /home/stig/go/src/github.com/platinasystems/go/elib/loop/event.go:118 +0x135
Apr 19 15:07:12 invader24 goes.vnetd[8382]: github.com/platinasystems/go/elib/loop.(*Loop).doEvent(0xc820192000, 0xc8208ee120)
Apr 19 15:07:12 invader24 goes.vnetd[8382]: /home/stig/go/src/github.com/platinasystems/go/elib/loop/event.go:133 +0x48
Apr 19 15:07:12 invader24 goes.vnetd[8382]: github.com/platinasystems/go/elib/loop.(*Loop).eventHandler(0xc820192000, 0x7f5330fe3608, 0xc820192548)
Apr 19 15:07:12 invader24 goes.vnetd[8382]: /home/stig/go/src/github.com/platinasystems/go/elib/loop/event.go:140 +0x81
Apr 19 15:07:12 invader24 goes.vnetd[8382]: created by github.com/platinasystems/go/elib/loop.(*Loop).startHandler
Apr 19 15:07:12 invader24 goes.vnetd[8382]: /home/stig/go/src/github.com/platinasystems/go/elib/loop/event.go:153 +0x120
Apr 19 15:07:12 invader24 goes.vnetd[8382]: done

Any vnet show cmd crashes vnet

sudo goes restart
ps -e f
sudo goes vnet show ip fib
[takes a long time, ~5s]
ps -e f
[vnet is dead]
syslog:
"runtime error: index out of range: goroutine 45 [running]"

Version:

commit c332849fbd55d6ef85c5e89c8803778622456916
Author: Mark Pleso <[email protected]>
Date:   Sat Feb 18 23:47:35 2017 -0800

    Changed mknod permissions to octal

    Signed-off-by: Mark Pleso <[email protected]>

Details:

donn@invader8:~/workspace/go/src/github.com/platinasystems/go$ sudo goes restart

donn@invader8:~/workspace/go/src/github.com/platinasystems/go$ ps -e f
  PID TTY      STAT   TIME COMMAND
    2 ?        S      0:00 [kthreadd]
    3 ?        S      0:57  \_ [ksoftirqd/0]
    5 ?        S<     0:00  \_ [kworker/0:0H]
<...>
18958 ?        Ssl    0:00 goes-daemons
18966 ?        Sl     0:00  \_ redisd
19002 ?        Sl     0:00  \_ vnetd
19003 ?        Sl     0:00  \_ uptimed
19007 ?        Sl     0:00  \_ nld

donn@invader8:~/workspace/go/src/github.com/platinasystems/go$ sudo goes vnet sho
w ip fib
vnet: /run/goes/socks/vnet: timeout
donn@invader8:~/workspace/go/src/github.com/platinasystems/go$

donn@invader8:~/workspace/go/src/github.com/platinasystems/go$ ps -e f
  PID TTY      STAT   TIME COMMAND
    2 ?        S      0:00 [kthreadd]
    3 ?        S      0:57  \_ [ksoftirqd/0]
    5 ?        S<     0:00  \_ [kworker/0:0H]
<...>
18958 ?        Ssl    0:00 goes-daemons
18966 ?        Sl     0:00  \_ redisd
19003 ?        Sl     0:00  \_ uptimed
19007 ?        Sl     0:00  \_ nld
donn@invader8:~/workspace/go/src/github.com/platinasystems/go$

syslog:


Feb 21 14:40:32 invader8 goes.redisd[18966]: listen: /run/goes/socks/redisd
Feb 21 14:40:32 invader8 goes.redisd[18966]: listen: [7.7.7.7%lo]:6379
Feb 21 14:40:32 invader8 goes.redisd[18966]: listen: [::1%lo]:6379
Feb 21 14:40:32 invader8 goes.redisd[18966]: listen: [192.168.101.128%eth0]:6379
Feb 21 14:40:32 invader8 goes.redisd[18966]: listen: [fe80::46:8aff:fe00:53e%eth0
]:6379
Feb 21 14:40:49 invader8 goes.vnetd[19002]: runtime error: index out of range: go
routine 45 [running]:
Feb 21 14:40:49 invader8 goes.vnetd[19002]: runtime/debug.Stack(0x0, 0x0, 0x0)
Feb 21 14:40:49 invader8 goes.vnetd[19002]:         /usr/local/go/src/runtime/deb
ug/stack.go:24 +0x80
Feb 21 14:40:49 invader8 goes.vnetd[19002]: github.com/platinasystems/go/elib/loo
p.(*Loop).doEvent.func1(0xc820184000)
Feb 21 14:40:49 invader8 goes.vnetd[19002]:         /home/jenkins/workspace/go/sr
c/github.com/platinasystems/go/elib/loop/event.go:128 +0xa2
Feb 21 14:40:49 invader8 goes.vnetd[19002]: panic(0xda3fa0, 0xc82000a030)
Feb 21 14:40:49 invader8 goes.vnetd[19002]:         /usr/local/go/src/runtime/pan
ic.go:443 +0x4e9
Feb 21 14:40:49 invader8 goes.vnetd[19002]: github.com/platinasystems/go/vnet/dev
ices/ethernet/switch/fe1/internal/fe1a.(*fe1a).ip4_fib_add_del(0xc820230000, 0xc8
00000000, 0xc8213167d8, 0x2)
Feb 21 14:40:49 invader8 goes.vnetd[19002]:         /home/jenkins/workspace/go/sr
c/github.com/platinasystems/go/vnet/devices/ethernet/switch/fe1/internal/fe1a/fib
.go:286 +0xf63
Feb 21 14:40:49 invader8 goes.vnetd[19002]: github.com/platinasystems/go/vnet/dev
ices/ethernet/switch/fe1/internal/fe1a.(*fe1a).(github.com/platinasystems/go/vnet
/devices/ethernet/switch/fe1/internal/fe1a.ip4_fib_add_del)-fm(0xc800000000, 0xc8
213167d8, 0x2)
Feb 21 14:40:49 invader8 goes.vnetd[19002]:         /home/jenkins/workspace/go/sr
c/github.com/platinasystems/go/vnet/devices/ethernet/switch/fe1/internal/fe1a/fib
.go:49 +0x4c
Feb 21 14:40:49 invader8 goes.vnetd[19002]: github.com/platinasystems/go/vnet/ip4.(*Fib).addDel(0xc8204fa160, 0xc8201f8000, 0xc8213167d8, 0x2, 0x7f01fffffffe)
Feb 21 14:40:49 invader8 goes.vnetd[19002]:         /home/jenkins/workspace/go/src/github.com/platinasystems/go/vnet/ip4/fib.go:217 +0x32f
Feb 21 14:40:49 invader8 goes.vnetd[19002]: github.com/platinasystems/go/vnet/ip4.(*Main).addDelRoute(0xc8201f8000, 0xc82130f940, 0x200000000, 0x0, 0xdbd4c0, 0x0, 0x0)
Feb 21 14:40:49 invader8 goes.vnetd[19002]:         /home/jenkins/workspace/go/src/github.com/platinasystems/go/vnet/ip4/fib.go:367 +0x1cf
Feb 21 14:40:49 invader8 goes.vnetd[19002]: github.com/platinasystems/go/vnet/ip4.(*Main).(github.com/platinasystems/go/vnet/ip4.addDelRoute)-fm(0xc82130f940, 0x200000000, 0x0, 0xc8201a59f8, 0x0, 0x0)
Feb 21 14:40:49 invader8 goes.vnetd[19002]:         /home/jenkins/workspace/go/src/github.com/platinasystems/go/vnet/ip4/package.go:43 +0x51
Feb 21 14:40:49 invader8 goes.vnetd[19002]: github.com/platinasystems/go/vnet/unix.(*Main).ip4IfaddrMsg(0xc8201ce480, 0xc820098420, 0x0, 0x0)
Feb 21 14:40:49 invader8 goes.vnetd[19002]:         /home/jenkins/workspace/go/src/github.com/platinasystems/go/vnet/unix/netlink.go:386 +0x3d4
Feb 21 14:40:49 invader8 goes.vnetd[19002]: github.com/platinasystems/go/vnet/unix.(*netlinkEvent).EventAction(0xc820504b10)
Feb 21 14:40:49 invader8 goes.vnetd[19002]:         /home/jenkins/workspace/go/src/github.com/platinasystems/go/vnet/unix/netlink.go:288 +0x2a1
Feb 21 14:40:49 invader8 goes.vnetd[19002]: github.com/platinasystems/go/elib/loop.(*loopEvent).do(0xc82088a000)
Feb 21 14:40:49 invader8 goes.vnetd[19002]:         /home/jenkins/workspace/go/src/github.com/platinasystems/go/elib/loop/event.go:118 +0x135
Feb 21 14:40:49 invader8 goes.vnetd[19002]: github.com/platinasystems/go/elib/loop.(*Loop).doEvent(0xc820184000, 0xc82088a000)
Feb 21 14:40:49 invader8 goes.vnetd[19002]:         /home/jenkins/workspace/go/src/github.com/platinasystems/go/elib/loop/event.go:133 +0x48
Feb 21 14:40:49 invader8 goes.vnetd[19002]: github.com/platinasystems/go/elib/loop.(*Loop).eventHandler(0xc820184000, 0x7f1a70120028, 0xc820184548)
Feb 21 14:40:49 invader8 goes.vnetd[19002]:         /home/jenkins/workspace/go/src/github.com/platinasystems/go/elib/loop/event.go:140 +0x81
Feb 21 14:40:49 invader8 goes.vnetd[19002]: created by github.com/platinasystems/go/elib/loop.(*Loop).startHandler
Feb 21 14:40:49 invader8 goes.vnetd[19002]:         /home/jenkins/workspace/go/src/github.com/platinasystems/go/elib/loop/event.go:153 +0x120

goes stop doesn't remove linux interfaces

goes stop doesn't remove the linux interfaces. "ip link show" still shows all port and subports.

In stop.go, kill.All happens before Hook (replaced by stopHook in main.go).
In stopHook, it's checking redis for deviceVersion but since redisd is already killed, returns error so rest of code doesn't get executed.

vnet devices are republishing old counts

Note eth-31-0.* are repating with same counts...

$ goes subscribe platina
eth0.rx_packets: 162505931
eth0.tx_packets: 141682625
eth0.rx_bytes: 17609416052
eth0.tx_bytes: 10567673774
eth-31-0.tx_pipe_multicast_queue_cos0_packets: 7
eth-31-0.tx_pipe_multicast_queue_cos0_bytes: 738
eth-31-0.port_tx_packets: 7
eth-31-0.port_tx_bytes: 766
eth-31-0.port_tx_65_to_127_byte_packets: 5
eth-31-0.port_tx_128_to_255_byte_packets: 2
eth-31-0.port_tx_good_packets: 7
eth-31-0.port_tx_multicast_packets: 7
eth-31-0.tx_packets: 7
eth-31-0.tx_bytes: 738
eth-31-0-unix.rx_packets: 7
eth-31-0-unix.rx_bytes: 738
eth0.rx_packets: 162506230
eth0.tx_packets: 141682899
eth0.rx_bytes: 17609445872
eth0.tx_bytes: 10567699716
eth-31-0.tx_pipe_multicast_queue_cos0_packets: 7
eth-31-0.tx_pipe_multicast_queue_cos0_bytes: 738
eth-31-0.port_tx_packets: 7
eth-31-0.port_tx_bytes: 766
eth-31-0.port_tx_65_to_127_byte_packets: 5
eth-31-0.port_tx_128_to_255_byte_packets: 2
eth-31-0.port_tx_good_packets: 7
eth-31-0.port_tx_multicast_packets: 7
eth-31-0.tx_packets: 7
eth-31-0.tx_bytes: 738
eth-31-0-unix.rx_packets: 7
eth-31-0-unix.rx_bytes: 738

Fib multipath for shared adjacency

In a namespace environment a learned route seems to be treated as a multipath:

root@invader5:/home/dlobete# goes vnet show ip fib
Table Destination Adjacency
0 31.0.0.0/24 5: glean eth-24-0
0 31.0.0.5/32 6: local eth-24-0
2 10.0.0.0/24 3: glean eth-1-0
2 10.0.0.1/32 4: local eth-1-0
tx pipe packets 2
tx pipe bytes 148
2 10.0.0.2/32 12: rewrite eth-1-0 IP4: 02:46:8a:00:04:36 -> 02:46:8a:00:04:37
2 30.0.0.0/24 9: glean eth-10-0
2 30.0.0.5/32 10: local eth-10-0
3 10.0.0.0/24 7: glean eth-2-0
3 10.0.0.1/32 11: rewrite eth-2-0 IP4: 02:46:8a:00:04:37 -> 02:46:8a:00:04:36
3 10.0.0.2/32 8: local eth-2-0
tx pipe packets 2
tx pipe bytes 148
3 30.0.0.0/24 13: rewrite eth-2-0 IP4: 02:46:8a:00:04:37 -> 02:46:8a:00:04:36 13-13, 1 x 11

Probably the last entry in table 3 for 30.0.0.0/24 should share the adjacency (11) instead of having its own adjacency (13).

Very intermittently on a teardown of the namespace I see a panic in vnet:

May 23 15:58:08 invader5 goes.vnetd[544]: runtime error: invalid memory address or nil pointer dereference: goroutine 39 [running]:
May 23 15:58:08 invader5 goes.vnetd[544]: runtime/debug.Stack(0xc42093b860, 0xadb800, 0x1201470)
May 23 15:58:08 invader5 goes.vnetd[544]: /usr/local/go/src/runtime/debug/stack.go:24 +0x79
May 23 15:58:08 invader5 goes.vnetd[544]: github.com/platinasystems/go/elib/loop.(*Loop).doEvent.func1(0xc420196000)
May 23 15:58:08 invader5 goes.vnetd[544]: /home/dlobete/src/github.com/platinasystems/go/elib/loop/event.go:128 +0x72
May 23 15:58:08 invader5 goes.vnetd[544]: panic(0xadb800, 0x1201470)
May 23 15:58:08 invader5 goes.vnetd[544]: /usr/local/go/src/runtime/panic.go:489 +0x2cf
May 23 15:58:08 invader5 goes.vnetd[544]: github.com/platinasystems/go/vnet/ip4.(*Fib).mapFibRemapAdjacency(0x0, 0xc420184380, 0xfffffffe0000000f)
May 23 15:58:08 invader5 goes.vnetd[544]: /home/dlobete/src/github.com/platinasystems/go/vnet/ip4/fib.go:252 +0x48
May 23 15:58:08 invader5 goes.vnetd[544]: github.com/platinasystems/go/vnet/ip4.(*Main).remapAdjacency(0xc420184380, 0xfffffffe0000000f)
May 23 15:58:08 invader5 goes.vnetd[544]: /home/dlobete/src/github.com/platinasystems/go/vnet/ip4/fib.go:273 +0x78
May 23 15:58:08 invader5 goes.vnetd[544]: github.com/platinasystems/go/vnet/ip4.(*Main).(github.com/platinasystems/go/vnet/ip4.remapAdjacency)-fm(0xfffffffe0000000f)
May 23 15:58:08 invader5 goes.vnetd[544]: /home/dlobete/src/github.com/platinasystems/go/vnet/ip4/package.go:25 +0x3a
May 23 15:58:08 invader5 goes.vnetd[544]: github.com/platinasystems/go/vnet/ip.(*Main).delMultipathAdj(0xc4201843e8, 0x10000000e)
May 23 15:58:08 invader5 goes.vnetd[544]: /home/dlobete/src/github.com/platinasystems/go/vnet/ip/adjacency.go:615 +0x1fc
May 23 15:58:08 invader5 goes.vnetd[544]: github.com/platinasystems/go/vnet/ip.(*Main).FreeAdj(0xc4201843e8, 0x10000000e)
May 23 15:58:08 invader5 goes.vnetd[544]: /home/dlobete/src/github.com/platinasystems/go/vnet/ip/adjacency.go:553 +0x65
May 23 15:58:08 invader5 goes.vnetd[544]: github.com/platinasystems/go/vnet/ip.(*Main).DelAdj(0xc4201843e8, 0xe0000000e)
May 23 15:58:08 invader5 goes.vnetd[544]: /home/dlobete/src/github.com/platinasystems/go/vnet/ip/adjacency.go:560 +0x4e
May 23 15:58:08 invader5 goes.vnetd[544]: github.com/platinasystems/go/vnet/ethernet.(*ipNeighborMain).AddDelIpNeighbor(0xc420162668, 0xc4201843e8, 0xc4213481a0, 0x2c04000001, 0x1, 0x0)
May 23 15:58:08 invader5 goes.vnetd[544]: /home/dlobete/src/github.com/platinasystems/go/vnet/ethernet/neighbor.go:90 +0x5a9
May 23 15:58:08 invader5 goes.vnetd[544]: github.com/platinasystems/go/vnet/unix.(*Main).ip4NeighborMsg(0xc4201e2500, 0xc4200ec540, 0x0, 0x0)
May 23 15:58:08 invader5 goes.vnetd[544]: /home/dlobete/src/github.com/platinasystems/go/vnet/unix/netlink.go:488 +0x322

panic: unix:/run/goes/socks/vnet -> unix:@

Beta hw invader (i19)

platina@invader19:~$ /usr/bin/goes version
acbcdb9

I configured autoneg and brought up a copper 100GE link to a pktgen/bcm-shell tor (i14).
I did nothing else.
After working on another project, I returned about 2h later and discovered vnet had died.

Jan 26 13:17:01 invader19 CRON[1624]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
Jan 26 13:42:01 invader19 goes.vnetd[455]: panic: unix:/run/goes/socks/vnet -> unix:@
Jan 26 13:42:01 invader19 goes.vnetd[455]:
Jan 26 13:42:01 invader19 goes.vnetd[455]: goroutine 7 [running]:
Jan 26 13:42:01 invader19 goes.vnetd[455]: panic(0xadeb20, 0xc42015f208)
Jan 26 13:42:01 invader19 goes.vnetd[455]: /usr/local/go/src/runtime/panic.go:500 +0x1a1
Jan 26 13:42:01 invader19 goes.vnetd[455]: github.com/platinasystems/go/elib/socket.(*socket).ErrorReady(0xc42015f208, 0x0, 0x0)
Jan 26 13:42:01 invader19 goes.vnetd[455]: /home/jpang/gopath/src/github.com/platinasystems/go/elib/socket/socket.go:221 +0x37
Jan 26 13:42:01 invader19 goes.vnetd[455]: github.com/platinasystems/go/elib/cli.(*File).ErrorReady(0xc4204ae7d0, 0x1211ac0, 0xc420f720f0)
Jan 26 13:42:01 invader19 goes.vnetd[455]: :59 +0x4a
Jan 26 13:42:01 invader19 goes.vnetd[455]: github.com/platinasystems/go/elib/iomux.(*Mux).do(0x123e760, 0xc420041350)
Jan 26 13:42:01 invader19 goes.vnetd[455]: /home/jpang/gopath/src/github.com/platinasystems/go/elib/iomux/epoll.go:153 +0x15d
Jan 26 13:42:01 invader19 goes.vnetd[455]: github.com/platinasystems/go/elib/iomux.(*Mux).EventPoll(0x123e760)
Jan 26 13:42:01 invader19 goes.vnetd[455]: /home/jpang/gopath/src/github.com/platinasystems/go/elib/iomux/epoll.go:166 +0xde
Jan 26 13:42:01 invader19 goes.vnetd[455]: github.com/platinasystems/go/elib/loop.(*eventLoop).eventPoller(0xc420184370, 0x1211c00, 0x123e760)
Jan 26 13:42:01 invader19 goes.vnetd[455]: /home/jpang/gopath/src/github.com/platinasystems/go/elib/loop/event.go:158 +0x31
Jan 26 13:42:01 invader19 goes.vnetd[455]: created by github.com/platinasystems/go/elib/loop.(*eventLoop).startPoller
Jan 26 13:42:01 invader19 goes.vnetd[455]: /home/jpang/gopath/src/github.com/platinasystems/go/elib/loop/event.go:161 +0x53
Jan 26 13:42:03 invader19 goes.vnetd[455]: exit status 2
Jan 26 14:17:01 invader19 CRON[1708]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)

These are the cmds I did:

77 sudo shutdown -h now
78 ps auxw | grep "vnet"
79 sudo goes vnet show fe1 po ph
80 hset platina eth-32-0.media copper
81 hset platina eth-32-0.speed auto
82 goes hset platina eth-32-0.media copper
83 sudo goes hset platina eth-32-0.media copper
84 sudo goes hset platina eth-32-1.media copper
85 sudo goes hset platina eth-32-1.speed auto
86 sudo goes vnet show fe1 po ph | grep eth-32
87 sudo goes vnet show fe1 po ph | head
88 sudo less /var/log/syslog
89 date
90 ps -e f
91 sudo less /var/log/syslog
92 sudo cp -ip /var/log/syslog ~/syslog_2017_01_26_panic_socket_op.txt

When I executed command #87, it said:

platina@invader19:~$ sudo goes vnet show fe1 po ph | head
[sudo] password for platina:
vnet: dial unix /run/goes/socks/vnet: connect: connection refused

This was at:

platina@invader19:~$ date
Thu Jan 26 15:12:42 PST 2017

But it seems vnet died earlier at 13:42

Syslog saved on i19:~platina/syslog_2017_01_26_panic_socket_op.txt

goes.qsfp fatal error: concurrent map read and map write

Saw this panic after a few loops for goes stop and start. 24 of the QSFP port connected to either another invader or pkgent; the other 8 ports are empty.

code version: fe1252f

Apr 27 12:12:06 invader2 goes.qsfp[31903]: fatal error: concurrent map read and map write
Apr 27 12:12:06 invader2 goes.qsfp[31903]:
Apr 27 12:12:06 invader2 goes.qsfp[31903]: goroutine 3 [running]:
Apr 27 12:12:06 invader2 goes.qsfp[31903]: runtime.throw(0xbbd183, 0x21)
Apr 27 12:12:06 invader2 goes.qsfp[31903]: /usr/local/go/src/runtime/panic.go:566 +0x95 fp=0xc420041cd0 sp=0xc420041cb0
Apr 27 12:12:06 invader2 goes.qsfp[31903]: runtime.mapaccess1_faststr(0xaa41e0, 0xc42014a3c0, 0xc42000e400, 0x15, 0x1)
Apr 27 12:12:06 invader2 goes.qsfp[31903]: /usr/local/go/src/runtime/hashmap_fast.go:201 +0x4f3 fp=0xc420041d30 sp=0xc420041cd0
Apr 27 12:12:06 invader2 goes.qsfp[31903]: github.com/platinasystems/go/internal/goes/cmd/qsfp.(*cmd).updateio(0xc42016c180, 0xc420188120, 0x0)
Apr 27 12:12:06 invader2 goes.qsfp[31903]: /home/fyang/gopath/src/github.com/platinasystems/go/internal/goes/cmd/qsfp/qsfpio.go:68 +0x26f fp=0xc420041eb0 sp
=0xc420041d30
Apr 27 12:12:06 invader2 goes.qsfp[31903]: github.com/platinasystems/go/internal/goes/cmd/qsfp.qsfpioTicker(0xc42016c180, 0x0, 0x0)
Apr 27 12:12:06 invader2 goes.qsfp[31903]: /home/fyang/gopath/src/github.com/platinasystems/go/internal/goes/cmd/qsfp/qsfpio.go:38 +0x143 fp=0xc420041f88 sp
=0xc420041eb0
Apr 27 12:12:06 invader2 goes.qsfp[31903]: runtime.goexit()
Apr 27 12:12:06 invader2 goes.qsfp[31903]: /usr/local/go/src/runtime/asm_amd64.s:2086 +0x1 fp=0xc420041f90 sp=0xc420041f88
Apr 27 12:12:06 invader2 goes.qsfp[31903]: created by github.com/platinasystems/go/internal/goes/cmd/qsfp.(*cmd).Main
Apr 27 12:12:06 invader2 goes.qsfp[31903]: /home/fyang/gopath/src/github.com/platinasystems/go/internal/goes/cmd/qsfp/qsfp.go:82 +0x254
Apr 27 12:12:06 invader2 goes.qsfp[31903]:
Apr 27 12:12:06 invader2 goes.qsfp[31903]: goroutine 1 [runnable]:
Apr 27 12:12:06 invader2 goes.qsfp[31903]: reflect.Value.FieldByIndex(0xafb680, 0xc420178fd8, 0x199, 0xc420071b08, 0x1, 0x1, 0x729552, 0x16, 0xc4201764e0)
Apr 27 12:12:06 invader2 goes.qsfp[31903]: /usr/local/go/src/reflect/value.go:771
Apr 27 12:12:06 invader2 goes.qsfp[31903]: encoding/gob.(*Encoder).encodeStruct(0xc42009fd60, 0xc42009fd98, 0xc4201764e0, 0xafb680, 0xc420178fd8, 0x199)
Apr 27 12:12:06 invader2 goes.qsfp[31903]: /usr/local/go/src/encoding/gob/encode.go:326 +0x20f
Apr 27 12:12:06 invader2 goes.qsfp[31903]: encoding/gob.(*Encoder).encode(0xc42009fd60, 0xc42009fd98, 0xafb680, 0xc420178fd8, 0x199, 0xc42006fa00)
Apr 27 12:12:06 invader2 goes.qsfp[31903]: /usr/local/go/src/encoding/gob/encode.go:707 +0x1db
Apr 27 12:12:06 invader2 goes.qsfp[31903]: encoding/gob.(*Encoder).EncodeValue(0xc42009fd60, 0xa2e4c0, 0xc420178fd8, 0x16, 0x0, 0x0)
Apr 27 12:12:06 invader2 goes.qsfp[31903]: /usr/local/go/src/encoding/gob/encoder.go:247 +0x3b0
Apr 27 12:12:06 invader2 goes.qsfp[31903]: encoding/gob.(*Encoder).Encode(0xc42009fd60, 0xa2e4c0, 0xc420178fd8, 0xc4201d8250, 0xc4201d8260)
Apr 27 12:12:06 invader2 goes.qsfp[31903]: /usr/local/go/src/encoding/gob/encoder.go:175 +0x61
Apr 27 12:12:06 invader2 goes.qsfp[31903]: net/rpc.(*gobClientCodec).WriteRequest(0xc4201a06c0, 0xc420178fd8, 0xa1aac0, 0x1319d60, 0x4000000044d575, 0xc4201d82c0)
Apr 27 12:12:06 invader2 goes.qsfp[31903]: /usr/local/go/src/net/rpc/client.go:213 +0x49
Apr 27 12:12:06 invader2 goes.qsfp[31903]: net/rpc.(*Client).send(0xc420178fc0, 0xc420149f40)
Apr 27 12:12:06 invader2 goes.qsfp[31903]: /usr/local/go/src/net/rpc/client.go:91 +0x1dd
Apr 27 12:12:06 invader2 goes.qsfp[31903]: net/rpc.(*Client).Go(0xc420178fc0, 0xbb042a, 0x10, 0xa1aac0, 0x1319d60, 0xa1ab00, 0x12f4280, 0xc4202944e0, 0x10)
Apr 27 12:12:06 invader2 goes.qsfp[31903]: /usr/local/go/src/net/rpc/client.go:309 +0xf2
Apr 27 12:12:06 invader2 goes.qsfp[31903]: net/rpc.(*Client).Call(0xc420178fc0, 0xbb042a, 0x10, 0xa1aac0, 0x1319d60, 0xa1ab00, 0x12f4280, 0x0, 0x0)
Apr 27 12:12:06 invader2 goes.qsfp[31903]: /usr/local/go/src/net/rpc/client.go:315 +0x97
Apr 27 12:12:06 invader2 goes.qsfp[31903]: github.com/platinasystems/go/internal/goes/cmd/qsfp.DoI2cRpc()
Apr 27 12:12:06 invader2 goes.qsfp[31903]: /home/fyang/gopath/src/github.com/platinasystems/go/internal/goes/cmd/qsfp/qsfp_rw.go:188 +0xb7
Apr 27 12:12:06 invader2 goes.qsfp[31903]: github.com/platinasystems/go/internal/goes/cmd/qsfp.(*I2cDev).SN(0x1317fa0, 0xba5ad4, 0x5)
Apr 27 12:12:06 invader2 goes.qsfp[31903]: /home/fyang/gopath/src/github.com/platinasystems/go/internal/goes/cmd/qsfp/qsfp.go:498 +0x5b
Apr 27 12:12:06 invader2 goes.qsfp[31903]: github.com/platinasystems/go/internal/goes/cmd/qsfp.(*cmd).update(0xc42016c180, 0xc4201880c0, 0x0)
Apr 27 12:12:06 invader2 goes.qsfp[31903]: /home/fyang/gopath/src/github.com/platinasystems/go/internal/goes/cmd/qsfp/qsfp.go:209 +0x32f5
Apr 27 12:12:06 invader2 goes.qsfp[31903]: github.com/platinasystems/go/internal/goes/cmd/qsfp.(*cmd).Main(0xc42016c180, 0xc420070160, 0x0, 0x0, 0x0, 0x0)
Apr 27 12:12:06 invader2 goes.qsfp[31903]: /home/fyang/gopath/src/github.com/platinasystems/go/internal/goes/cmd/qsfp/qsfp.go:90 +0x374
Apr 27 12:12:06 invader2 goes.qsfp[31903]: github.com/platinasystems/go/internal/goes.(Cmd).Main-fm(0xc420070160, 0x0, 0x0, 0xc420178180, 0xc420045d10)
Apr 27 12:12:06 invader2 goes.qsfp[31903]: /home/fyang/gopath/src/github.com/platinasystems/go/internal/goes/goes.go:229 +0x4d
Apr 27 12:12:06 invader2 goes.qsfp[31903]: github.com/platinasystems/go/internal/goes.ByName.Main(0xc42016c150, 0xc420070160, 0x0, 0x0, 0x0, 0x0)
Apr 27 12:12:06 invader2 goes.qsfp[31903]: /home/fyang/gopath/src/github.com/platinasystems/go/internal/goes/goes.go:202 +0x588
Apr 27 12:12:06 invader2 goes.qsfp[31903]: main.main()
Apr 27 12:12:06 invader2 goes.qsfp[31903]: net/rpc.(*gobClientCodec).WriteRequest(0xc4201a06c0, 0xc420178fd8, 0xa1aac0, 0x1319d60, 0x4000000044d575, 0xc4201d82c0)
Apr 27 12:12:06 invader2 goes.qsfp[31903]: /usr/local/go/src/net/rpc/client.go:213 +0x49
Apr 27 12:12:06 invader2 goes.qsfp[31903]: net/rpc.(*Client).send(0xc420178fc0, 0xc420149f40)
Apr 27 12:12:06 invader2 goes.qsfp[31903]: /usr/local/go/src/net/rpc/client.go:91 +0x1dd
Apr 27 12:12:06 invader2 goes.qsfp[31903]: net/rpc.(*Client).Go(0xc420178fc0, 0xbb042a, 0x10, 0xa1aac0, 0x1319d60, 0xa1ab00, 0x12f4280, 0xc4202944e0, 0x10)
Apr 27 12:12:06 invader2 goes.qsfp[31903]: /usr/local/go/src/net/rpc/client.go:309 +0xf2
Apr 27 12:12:06 invader2 goes.qsfp[31903]: net/rpc.(*Client).Call(0xc420178fc0, 0xbb042a, 0x10, 0xa1aac0, 0x1319d60, 0xa1ab00, 0x12f4280, 0x0, 0x0)
Apr 27 12:12:06 invader2 goes.qsfp[31903]: /usr/local/go/src/net/rpc/client.go:315 +0x97
Apr 27 12:12:06 invader2 goes.qsfp[31903]: github.com/platinasystems/go/internal/goes/cmd/qsfp.DoI2cRpc()
Apr 27 12:12:06 invader2 goes.qsfp[31903]: /home/fyang/gopath/src/github.com/platinasystems/go/internal/goes/cmd/qsfp/qsfp_rw.go:188 +0xb7
Apr 27 12:12:06 invader2 goes.qsfp[31903]: github.com/platinasystems/go/internal/goes/cmd/qsfp.(*I2cDev).SN(0x1317fa0, 0xba5ad4, 0x5)
Apr 27 12:12:06 invader2 goes.qsfp[31903]: /home/fyang/gopath/src/github.com/platinasystems/go/internal/goes/cmd/qsfp/qsfp.go:498 +0x5b
Apr 27 12:12:06 invader2 goes.qsfp[31903]: github.com/platinasystems/go/internal/goes/cmd/qsfp.(*cmd).update(0xc42016c180, 0xc4201880c0, 0x0)
Apr 27 12:12:06 invader2 goes.qsfp[31903]: /home/fyang/gopath/src/github.com/platinasystems/go/internal/goes/cmd/qsfp/qsfp.go:209 +0x32f5
Apr 27 12:12:06 invader2 goes.qsfp[31903]: github.com/platinasystems/go/internal/goes/cmd/qsfp.(*cmd).Main(0xc42016c180, 0xc420070160, 0x0, 0x0, 0x0, 0x0)
Apr 27 12:12:06 invader2 goes.qsfp[31903]: /home/fyang/gopath/src/github.com/platinasystems/go/internal/goes/cmd/qsfp/qsfp.go:90 +0x374
Apr 27 12:12:06 invader2 goes.qsfp[31903]: github.com/platinasystems/go/internal/goes.(Cmd).Main-fm(0xc420070160, 0x0, 0x0, 0xc420178180, 0xc420045d10)
Apr 27 12:12:06 invader2 goes.qsfp[31903]: /home/fyang/gopath/src/github.com/platinasystems/go/internal/goes/goes.go:229 +0x4d
Apr 27 12:12:06 invader2 goes.qsfp[31903]: github.com/platinasystems/go/internal/goes.ByName.Main(0xc42016c150, 0xc420070160, 0x0, 0x0, 0x0, 0x0)
Apr 27 12:12:06 invader2 goes.qsfp[31903]: /home/fyang/gopath/src/github.com/platinasystems/go/internal/goes/goes.go:202 +0x588
Apr 27 12:12:06 invader2 goes.qsfp[31903]: main.main()
Apr 27 12:12:06 invader2 goes.qsfp[31903]: /home/fyang/gopath/src/github.com/platinasystems/go/main/goes-platina-mk1/main.go:14 +0x4c
Apr 27 12:12:06 invader2 goes.qsfp[31903]:
Apr 27 12:12:06 invader2 goes.qsfp[31903]: goroutine 17 [syscall, locked to thread]:
Apr 27 12:12:06 invader2 goes.qsfp[31903]: runtime.goexit()
Apr 27 12:12:06 invader2 goes.qsfp[31903]: /usr/local/go/src/runtime/asm_amd64.s:2086 +0x1
Apr 27 12:12:06 invader2 goes.qsfp[31903]:
Apr 27 12:12:06 invader2 goes.qsfp[31903]: goroutine 20 [syscall]:
Apr 27 12:12:06 invader2 goes.qsfp[31903]: os/signal.signal_recv(0x0)
Apr 27 12:12:06 invader2 goes.qsfp[31903]: /usr/local/go/src/runtime/sigqueue.go:116 +0x157
Apr 27 12:12:06 invader2 goes.qsfp[31903]: os/signal.loop()
Apr 27 12:12:06 invader2 goes.qsfp[31903]: /usr/local/go/src/os/signal/signal_unix.go:22 +0x22
Apr 27 12:12:06 invader2 goes.qsfp[31903]: created by os/signal.init.1
Apr 27 12:12:06 invader2 goes.qsfp[31903]: /usr/local/go/src/os/signal/signal_unix.go:28 +0x41
Apr 27 12:12:06 invader2 goes.qsfp[31903]:
Apr 27 12:12:06 invader2 goes.qsfp[31903]: goroutine 22 [select, locked to thread]:
Apr 27 12:12:06 invader2 goes.qsfp[31903]: runtime.gopark(0xc21498, 0x0, 0xba6ff2, 0x6, 0x18, 0x2)
Apr 27 12:12:06 invader2 goes.qsfp[31903]: /usr/local/go/src/runtime/proc.go:259 +0x13a
Apr 27 12:12:06 invader2 goes.qsfp[31903]: runtime.selectgoImpl(0xc42001df30, 0x0, 0x18)
Apr 27 12:12:06 invader2 goes.qsfp[31903]: /usr/local/go/src/runtime/select.go:423 +0x11d9
Apr 27 12:12:06 invader2 goes.qsfp[31903]: runtime.selectgo(0xc42001df30)
Apr 27 12:12:06 invader2 goes.qsfp[31903]: /usr/local/go/src/runtime/select.go:238 +0x1c
Apr 27 12:12:06 invader2 goes.qsfp[31903]: runtime.ensureSigM.func1()
Apr 27 12:12:06 invader2 goes.qsfp[31903]: /usr/local/go/src/runtime/signal1_unix.go:304 +0x2f3
Apr 27 12:12:06 invader2 goes.qsfp[31903]: runtime.goexit()
Apr 27 12:12:06 invader2 goes.qsfp[31903]: /usr/local/go/src/runtime/asm_amd64.s:2086 +0x1
Apr 27 12:12:06 invader2 goes.qsfp[31903]:
Apr 27 12:12:06 invader2 goes.qsfp[31903]: goroutine 23 [chan receive]:
Apr 27 12:12:06 invader2 goes.qsfp[31903]: github.com/platinasystems/go/internal/goes.(*Goes).wait(0xc42006d7a0, 0xc420178180)
Apr 27 12:12:06 invader2 goes.qsfp[31903]: /home/fyang/gopath/src/github.com/platinasystems/go/internal/goes/goes.go:262 +0x70
Apr 27 12:12:06 invader2 goes.qsfp[31903]: created by github.com/platinasystems/go/internal/goes.ByName.Main
Apr 27 12:12:06 invader2 goes.qsfp[31903]: /home/fyang/gopath/src/github.com/platinasystems/go/internal/goes/goes.go:200 +0x868
Apr 27 12:12:06 invader2 goes.qsfp[31903]:
Apr 27 12:12:06 invader2 goes.qsfp[31903]: goroutine 25 [IO wait]:
Apr 27 12:12:06 invader2 goes.qsfp[31903]: net.runtime_pollWait(0x7f9fc4910e08, 0x72, 0x5)
Apr 27 12:12:06 invader2 goes.qsfp[31903]: /usr/local/go/src/runtime/netpoll.go:160 +0x59
Apr 27 12:12:06 invader2 goes.qsfp[31903]: net.(*pollDesc).wait(0xc420068840, 0x72, 0xc420043a40, 0xc4200700a0)
Apr 27 12:12:06 invader2 goes.qsfp[31903]: /usr/local/go/src/net/fd_poll_runtime.go:73 +0x38
Apr 27 12:12:06 invader2 kernel: ixgbevf 0000:03:10.6: NIC Link is Up 10 Gbps
Apr 27 12:12:06 invader2 kernel: IPv6: ADDRCONF(NETDEV_UP): eth-3-0: link is not ready
Apr 27 12:12:06 invader2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth-3-0: link becomes ready
Apr 27 12:12:06 invader2 goes.qsfp[31903]: net.(*pollDesc).waitRead(0xc420068840, 0x12d2f40, 0xc4200700a0)
Apr 27 12:12:06 invader2 goes.qsfp[31903]: /usr/local/go/src/net/fd_poll_runtime.go:78 +0x34
Apr 27 12:12:06 invader2 goes.qsfp[31903]: net.(*netFD).Read(0xc4200687e0, 0xc420197000, 0x1000, 0x1000, 0x0, 0x12d2f40, 0xc4200700a0)
Apr 27 12:12:06 invader2 goes.qsfp[31903]: /usr/local/go/src/net/fd_unix.go:243 +0x1a1
Apr 27 12:12:06 invader2 goes.qsfp[31903]: net.(*conn).Read(0xc420078270, 0xc420197000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
Apr 27 12:12:06 invader2 goes.qsfp[31903]: /usr/local/go/src/net/net.go:173 +0x70
Apr 27 12:12:06 invader2 goes.qsfp[31903]: bufio.(*Reader).fill(0xc4201783c0)
Apr 27 12:12:06 invader2 goes.qsfp[31903]: /usr/local/go/src/bufio/bufio.go:97 +0x10c
Apr 27 12:12:06 invader2 goes.qsfp[31903]: bufio.(*Reader).Read(0xc4201783c0, 0xc420071930, 0x1, 0x9, 0xc420090600, 0x12dfc40, 0xa63da0)
Apr 27 12:12:06 invader2 goes.qsfp[31903]: /usr/local/go/src/bufio/bufio.go:209 +0x1bc
Apr 27 12:12:06 invader2 goes.qsfp[31903]: io.ReadAtLeast(0x12cf280, 0xc4201783c0, 0xc420071930, 0x1, 0x9, 0x1, 0xc4200501c0, 0xc420043c58, 0x45ab60)
Apr 27 12:12:06 invader2 goes.qsfp[31903]: /usr/local/go/src/io/io.go:307 +0xa4
Apr 27 12:12:06 invader2 goes.qsfp[31903]: io.ReadFull(0x12cf280, 0xc4201783c0, 0xc420071930, 0x1, 0x9, 0xc420043c80, 0x45ac70, 0xc420001380)
Apr 27 12:12:06 invader2 goes.qsfp[31903]: /usr/local/go/src/io/io.go:325 +0x58
Apr 27 12:12:06 invader2 goes.qsfp[31903]: encoding/gob.decodeUintReader(0x12cf280, 0xc4201783c0, 0xc420071930, 0x9, 0x9, 0x7ff297, 0xc420043d38, 0xc420043d48, 0x45aa60)
Apr 27 12:12:06 invader2 goes.qsfp[31903]: /usr/local/go/src/encoding/gob/decode.go:119 +0x63
Apr 27 12:12:06 invader2 goes.qsfp[31903]: encoding/gob.(*Decoder).recvMessage(0xc420090600, 0xc420043d38)
Apr 27 12:12:06 invader2 goes.qsfp[31903]: /usr/local/go/src/encoding/gob/decoder.go:76 +0x57
Apr 27 12:12:06 invader2 goes.qsfp[31903]: encoding/gob.(*Decoder).decodeTypeSequence(0xc420090600, 0xc21500, 0xc420090600)
Apr 27 12:12:06 invader2 goes.qsfp[31903]: /usr/local/go/src/encoding/gob/decoder.go:140 +0x16f
Apr 27 12:12:06 invader2 goes.qsfp[31903]: encoding/gob.(*Decoder).DecodeValue(0xc420090600, 0xa2e500, 0xc42014a450, 0x16, 0x0, 0x0)
Apr 27 12:12:06 invader2 goes.qsfp[31903]: /usr/local/go/src/encoding/gob/decoder.go:208 +0xd8
Apr 27 12:12:06 invader2 goes.qsfp[31903]: encoding/gob.(*Decoder).Decode(0xc420090600, 0xa2e500, 0xc42014a450, 0x0, 0x0)
Apr 27 12:12:06 invader2 goes.qsfp[31903]: /usr/local/go/src/encoding/gob/decoder.go:185 +0x16d
Apr 27 12:12:06 invader2 goes.qsfp[31903]: net/rpc.(*gobClientCodec).ReadResponseHeader(0xc42016d980, 0xc42014a450, 0x12f4920, 0x0)
Apr 27 12:12:06 invader2 goes.qsfp[31903]: /usr/local/go/src/net/rpc/client.go:223 +0x45
Apr 27 12:12:06 invader2 goes.qsfp[31903]: net/rpc.(*Client).input(0xc420178480)
Apr 27 12:12:06 invader2 goes.qsfp[31903]: /usr/local/go/src/net/rpc/client.go:109 +0xbd
Apr 27 12:12:06 invader2 goes.qsfp[31903]: created by net/rpc.NewClientWithCodec
Apr 27 12:12:06 invader2 goes.qsfp[31903]: /usr/local/go/src/net/rpc/client.go:201 +0xe4
Apr 27 12:12:06 invader2 goes.qsfp[31903]:
Apr 27 12:12:06 invader2 goes.qsfp[31903]: goroutine 26 [IO wait]:
Apr 27 12:12:06 invader2 goes.qsfp[31903]: net.runtime_pollWait(0x7f9fc4910d48, 0x72, 0x6)
Apr 27 12:12:06 invader2 goes.qsfp[31903]: /usr/local/go/src/runtime/netpoll.go:160 +0x59
Apr 27 12:12:06 invader2 goes.qsfp[31903]: net.(*pollDesc).wait(0xc4201941b0, 0x72, 0xc4201f1a40, 0xc4200700a0)
Apr 27 12:12:06 invader2 goes.qsfp[31903]: /usr/local/go/src/net/fd_poll_runtime.go:73 +0x38
Apr 27 12:12:06 invader2 goes.qsfp[31903]: net.(*pollDesc).waitRead(0xc4201941b0, 0x12d2f40, 0xc4200700a0)
Apr 27 12:12:06 invader2 goes.qsfp[31903]: /usr/local/go/src/net/fd_poll_runtime.go:78 +0x34
Apr 27 12:12:06 invader2 goes.qsfp[31903]: net.(*netFD).Read(0xc420194150, 0xc4201e6000, 0x1000, 0x1000, 0x0, 0x12d2f40, 0xc4200700a0)
Apr 27 12:12:06 invader2 goes.qsfp[31903]: /usr/local/go/src/net/fd_unix.go:243 +0x1a1
Apr 27 12:12:06 invader2 goes.qsfp[31903]: net.(*conn).Read(0xc420078540, 0xc4201e6000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
Apr 27 12:12:06 invader2 goes.qsfp[31903]: /usr/local/go/src/net/net.go:173 +0x70
Apr 27 12:12:06 invader2 goes.qsfp[31903]: bufio.(*Reader).fill(0xc420178f00)
Apr 27 12:12:06 invader2 goes.qsfp[31903]: /usr/local/go/src/bufio/bufio.go:97 +0x10c
Apr 27 12:12:06 invader2 goes.qsfp[31903]: bufio.(*Reader).Read(0xc420178f00, 0xc4201a5ee0, 0x1, 0x9, 0xc420090880, 0x12dfc40, 0xa63da0)
Apr 27 12:12:06 invader2 goes.qsfp[31903]: /usr/local/go/src/bufio/bufio.go:209 +0x1bc
Apr 27 12:12:06 invader2 goes.qsfp[31903]: io.ReadAtLeast(0x12cf280, 0xc420178f00, 0xc4201a5ee0, 0x1, 0x9, 0x1, 0xc42006f980, 0xc4201f1c58, 0x45ab60)
Apr 27 12:12:06 invader2 goes.qsfp[31903]: /usr/local/go/src/io/io.go:307 +0xa4
Apr 27 12:12:06 invader2 goes.qsfp[31903]: io.ReadFull(0x12cf280, 0xc420178f00, 0xc4201a5ee0, 0x1, 0x9, 0xc4201f1c80, 0x45ac70, 0xc4200001a0)
Apr 27 12:12:06 invader2 goes.qsfp[31903]: /usr/local/go/src/io/io.go:325 +0x58
Apr 27 12:12:06 invader2 goes.qsfp[31903]: encoding/gob.decodeUintReader(0x12cf280, 0xc420178f00, 0xc4201a5ee0, 0x9, 0x9, 0x7ff297, 0xc4201f1d38, 0xc4201f1d48, 0x45aa60)
Apr 27 12:12:06 invader2 goes.qsfp[31903]: /usr/local/go/src/encoding/gob/decode.go:119 +0x63
Apr 27 12:12:06 invader2 goes.qsfp[31903]: encoding/gob.(*Decoder).recvMessage(0xc420090880, 0xc4201f1d38)
Apr 27 12:12:06 invader2 goes.qsfp[31903]: /usr/local/go/src/encoding/gob/decoder.go:76 +0x57
Apr 27 12:12:06 invader2 goes.qsfp[31903]: encoding/gob.(*Decoder).decodeTypeSequence(0xc420090880, 0xc21500, 0xc420090880)
Apr 27 12:12:06 invader2 goes.qsfp[31903]: /usr/local/go/src/encoding/gob/decoder.go:140 +0x16f
Apr 27 12:12:06 invader2 goes.qsfp[31903]: encoding/gob.(*Decoder).DecodeValue(0xc420090880, 0xa2e500, 0xc4201a0720, 0x16, 0x0, 0x0)
Apr 27 12:12:06 invader2 goes.qsfp[31903]: /usr/local/go/src/encoding/gob/decoder.go:208 +0xd8
Apr 27 12:12:06 invader2 goes.qsfp[31903]: encoding/gob.(*Decoder).Decode(0xc420090880, 0xa2e500, 0xc4201a0720, 0x0, 0x0)
Apr 27 12:12:06 invader2 goes.qsfp[31903]: /usr/local/go/src/encoding/gob/decoder.go:185 +0x16d
Apr 27 12:12:06 invader2 goes.qsfp[31903]: net/rpc.(*gobClientCodec).ReadResponseHeader(0xc4201a06c0, 0xc4201a0720, 0x12f4280, 0x0)
Apr 27 12:12:06 invader2 goes.qsfp[31903]: /usr/local/go/src/net/rpc/client.go:223 +0x45
Apr 27 12:12:06 invader2 goes.qsfp[31903]: net/rpc.(*Client).input(0xc420178fc0)
Apr 27 12:12:06 invader2 goes.qsfp[31903]: /usr/local/go/src/net/rpc/client.go:109 +0xbd
Apr 27 12:12:06 invader2 goes.qsfp[31903]: created by net/rpc.NewClientWithCodec
Apr 27 12:12:06 invader2 goes.qsfp[31903]: /usr/local/go/src/net/rpc/client.go:201 +0xe4
Apr 27 12:12:06 invader2 goes.qsfp[31903]: exit status 2

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.