Client is stuck in the readLoop about nats.go

[derekcollison]

Will take a look, also will look at the CB ordering issue.

On Thu, Oct 23, 2014 at 10:06 PM, Michael Fraenkel <[email protected]

wrote:

Looks like we might need to set a read deadline in the client. All we know
is that this client is dead to NATs but its not budging off the read.

goroutine 55 [IO wait, 317 minutes]:
net.runtime_pollWait(0x7f4827bc75f8, 0x72, 0x0)
/usr/local/go/src/pkg/runtime/netpoll.goc:146 +0x66
net.(_pollDesc).Wait(0xc208085100, 0x72, 0x0, 0x0)
/usr/local/go/src/pkg/net/fd_poll_runtime.go:84 +0x46
net.(_pollDesc).WaitRead(0xc208085100, 0x0, 0x0)
/usr/local/go/src/pkg/net/fd_poll_runtime.go:89 +0x42
net.(_netFD).Read(0xc2080850a0, 0xc208122000, 0x8000, 0x8000, 0x0,
0x7f4827bc6418, 0xb)
/usr/local/go/src/pkg/net/fd_unix.go:242 +0x34c
net.(_conn).Read(0xc20802c120, 0xc208122000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
/usr/local/go/src/pkg/net/net.go:122 +0xe7
github.com/apcera/nats.(_Conn).readLoop(0xc20802ec00)
/var/vcap/packages/stager/src/github.com/apcera/nats/nats.go:865 +0x24c
created by github.com/apcera/nats.(_Conn).spinUpSocketWatchers
/var/vcap/packages/stager/src/github.com/apcera/nats/nats.go:428 +0x9f

—
Reply to this email directly or view it on GitHub
#34.

[derekcollison]

How do you know the connection is dead? Would this be better served with client side pings like gnatsd does?

Is this a result of aws machines just going away?

[fraenkel]

The Client Ping would be the ideal solution.
This is the only outstanding read for the NATs client. If there is a functional server ping, this should have been blocked for no more than 2 minutes. As you can see we have been here for 317 minutes. The aws machines don't actually go away, we just have been noticing that the connections get into an odd state, closer to some router which just closes the TCP connection without flowing the proper TCP packets.

[derekcollison]

I like the ping approach better since it does not assume anything about the
server side where the read timeouts, if that was treated as an error, are
assuming a server side ping interval.

On Thu, Oct 30, 2014 at 11:57 PM, Michael Fraenkel <[email protected]

wrote:

The Client Ping would be the ideal solution.
This is the only outstanding read for the NATs client. If there is a
functional server ping, this should have been blocked for no more than 2
minutes. As you can see we have been here for 317 minutes. The aws machines
don't actually go away, we just have been noticing that the connections get
into an odd state, closer to some router which just closes the TCP
connection without flowing the proper TCP packets.

—
Reply to this email directly or view it on GitHub
#34 (comment).