Expose ability to reconnect about nats.go

[krobertson]

How are you using the nats client? It will reconnect out of the box, however be aware it has a few reconnect settings you may want to switch, like MaxReconnect and ReconnectWait. If you are having an issue, you may be hitting the default max reconnect number.

[krobertson]

One other thing to add... nats behind HAProxy, are you running it in clustered mode? If not, it will produce some odd issues as not all clients will be on the same system. But with clustered, haproxy really isn't necessary. You can simply configure the clients with the list of all the NATS servers, and have them pick a random one rather than just the first one.

[chendo]

We want to have the ability to reconnect so we can rebalance the connections to another gnatsd server. A disconnect won't preserve the subscriptions.

We are running it in clustered mode. I have a fork that lets us dynamically add and remove routes to the cluster, so we can dynamically scale gnatsd up and down. HAProxy does the balancing and the config is automatically generated and updated via Marathon.

The problem comes when the machines that gnatsd are on die for some reason, and the clients all reconnect to the existing one, but when the replacements are spun back up, the connections are unbalanced to the one that survived.

[krobertson]

Were you hitting scaling limits with gnatsd? Although rebalancing can help and it avoids clumping on a single gnatsd server, can also be clumped or gated with haproxy itself.

While haproxy just proxies data, it may be that the latency from the added network hop ends ip costing more than an unbalanced set of clients.

[chendo]

Not yet, but I suspect we will at some point.

Our setup has haproxy running on every Mesos worker, so haproxy is always
on localhost, then it picks the least connections for a backend. This
doesn't guarantee that it gets sent to the least loaded backend as haproxy
doesn't share state with others but in our tests, it has been good enough.

On Saturday, March 7, 2015, Ken Robertson [email protected] wrote:

Were you hitting scaling limits with gnatsd? Although rebalancing can help
and it avoids clumping on a single gnatsd server, can also be clumped or
gated with haproxy itself.

While haproxy just proxies data, it may be that the latency from the added
network hop ends ip costing more than an unbalanced set of clients.

—
Reply to this email directly or view it on GitHub
#50 (comment).

Sent from my iPhone.

[derekcollison]

We will add rebalancing via gnatsd at some point in the future, most likely when we introduce the gossip protocol for clustering that will also transfer to the clients.