nats-io / nats.js Goto Github PK

What is the policy in regards to argument validation? Right now a lot of weird stuff can quietly happen or crash the entire node process if users pass in invalid arguments, but maybe we leave the responsibility to the user?

Right now the timeout method simply returns if there is nothing to timeout. Though, when that happens it is very likely a user side bug. Shouldn't we be a bit more verbose about it? Even if we don't throw an exception maybe at least a warning? I mean, e.g. passing sid==null doesn't make much sense.

Because Client.pongs is reset to null when a connection is closed, if the server dies or gets lost after a successful connection and you send a publish with an attached callback, the client will crash because it cannot push a callback onto null.

Example:

var nats = require('nats');

var nc = nats.connect();

nc.on('error', function(err) {
   console.log("Error:",err);
});

setInterval(function() {
   nc.publish('fee', 'bae', 'boison', function(msg) {
      console.log("Response: ",msg);
   });
},1000);

Let the client get a few pings then kill the server. The client will then die with:

test-nats/node_modules/nats/lib/nats.js:786
    this.pongs.push(opt_callback);
              ^
TypeError: Cannot read property 'push' of null

nats.subscribe('foo.bar.*', function(msg, reply, subject) {
console.log('Msg received on [' + subject + '] : ' + msg);
fnABC()
//if exception happened in fnABC() , the nats client will behave strange
//the msg will be random string
//or the subscribe can't receive any furture incoming msg
});

Right now if there is a pending flush, commands sent in the same tick as ´close` might fail? Optionally maybe flush before close? I'm not quite sure what would be the proper behaviour.

node-nats should support the same connection string as NATS. Currently, passing the username and password in the connection string is not supported.

nats://user:password@host:port

When the stream is closed, the client.inbound is not cleared. This means that when the client reconnects, it starts with a dirty buffer onto which new stream data chunks are appended. Consequently, if the stream is closed in the middle of receiving a message, parsing will fail when the stream reconnects. See here: abstractvector@3c274d0

node_modules/nats/lib/nats.js:447
        if (psize >= client.inbound.length) {
                                   ^
TypeError: Cannot read property 'length' of null
    at Socket.<anonymous> (/node_modules/nats/lib/nats.js:447:36)
    at emitOne (events.js:77:13)
    at Socket.emit (events.js:169:7)
    at readableAddChunk (_stream_readable.js:146:16)
    at Socket.Readable.push (_stream_readable.js:110:10)
    at TCP.onread (net.js:523:20)

Possibly user error? Needs some more error checking though as this doesn't give much to go on.

TypeError: Cannot read property '3' of null
    at Client.unsubscribe (/node_modules/nats/lib/nats.js:740:22)
    at [object Object].action (/src/lib/bus.js:53:24)

Probably user error but would be nice with better error messages.

We should have proper library API documentation showing in detail how we can use all the provided methods.

Currently all we can do is look into the source code. I haven't found anything on the official NATS website indicating all the clients should have similar API or something, so I cannot assume anything.

In example: I just realized there's a 'json' field that can be passed to 'connect' via source code.

Edit: Seems like the source code does have a documentation but it's only there and nowhere else (compiled). Is that a YUIDoc or something?

If a user passes in an invalid msg argument the publish function might throw an exception which will leave the subscription dangling. Request should probably look like this instead:

Client.prototype.request = function(subject, opt_msg, opt_options, callback) {
  if (typeof opt_msg === 'function') {
    callback = opt_msg;
    opt_msg = EMPTY;
    opt_options = null;
  }
  if (typeof opt_options === 'function') {
    callback = opt_options;
    opt_options = null;
  }
  var inbox = createInbox();
  this.publish(subject, opt_msg, inbox);
  return this.subscribe(inbox, opt_options, function(msg, reply) {
    callback(msg, reply);
  });
};

Also publish should probably assert that the msg is a string or explicitly try to convert it to a string since nothing else should work with Buffer.byteLength.

When I try to do:

nats.connect({
    url: '127.0.0.1',
    port: '4222'
  });

I get:

TypeError: port should be a number or string: null
    at Socket.connect (net.js:914:13)
    at Object.exports.connect.exports.createConnection (net.js:92:35)

It looks like there is no default for opts.port and opts.port is not being assigned as an option.

The createInbox implementation is a bit inefficient both in terms of size and uniqueness, I would suggest a much better implementation such as https://github.com/ericelliott/cuid.

Shouldn't we protect subscription callback invocations in a try/catch? Right now if the user provided callback throws an exception the parsing state becomes broken.

I need various browser clients to participate in a pub / sub style topology.
I am a golang programmer and so was wondering if there is a project to expose nats pub / sub to browser clients over any of the usual suspects such as websickets, long poll, SSE ?

The reason I prefer not to go with a nodejs approach is many, but mostly because I need to run on arm v6, v7 and v8 on Linux and windows.

It's of course perfectly possible that the js code in this repo can be used by browsers. I would then just need to write a golang server to expose the nats events over websickets. The js code would then be a proxy client to the golang server.
This is exactly how I would like to architect this, and so I am wondering if anyone could give me some opinions on this before I start coding it.

I am running gnatsd 0.8.0 (ARM version) and the nats 0.6.2 client for Node.js on Raspberry Pi 3. It works great.

My application publishes IoT sensor data messages that are only meaningful in real time. But if gnatsd is not running for an interval, the client queues published messages and then delivers them in a burst when it is next able to connect to gnatsd. I need to prevent that burst delivery of accumulated stale sensor data.

As an application level workaround, I can track connected true/false state based on the connected, reconnected, disconnected, and closed events and only publish if connected is true. But that requires wrapper logic for every call to publish and a global state variable. It gets a bit messy.

Or I can reach into the nats object and only publish if nats.connected is true. That eliminates the global state variable, but assumes a nats internal state variable that is not exported in the official API.

It might be useful if there were a new global option for nats.connect() and/or a per invocation option for nats.publish() where the application could specify the message should be discarded if gnatsd is not currently connected.

Have I missed something? Is there already a way to do this?

Thanks for the excellent open source NATS software.

Is there any chance of seeing it in node-nats?
And if so how would it be implemented?

I see that after using nats.request, the inbox subscription still exists, both on the server, and the client (nats.subs), probably a missing call to unsubscribe after processing the received message.

Happens when we switch between buffers and strings in terms of what the streams are handing us.

As far as I know all javascript platforms now support Promises natively. I would like to suggest that we upgrade the nats api to make use of that. Currently it doesn't even properly follow the node thunk pattern.

There should be a way to cancel a timeout request.

We are seeing some significant performance issues with this library when processing a lot of mesages arriving in bulk.

I'm seeing our event loop being blocked for 200-1200ms. Which is not quite acceptable.

The first thing we kind of need is to make sure that the nats parser yields back to the event loop so that it doesn't block other things from making progress, i.e. use setImmediate, setTimeout or process.nextTick after every nth message or some other time based heuristic.

Next thing is that the parser seems to be a bit slow, in relative terms.

I don't have time to give better metrics right now. I will come back to this issue later. Our current solution is to run node-nats in a web-worker.

@derekcollison Thanks for making nats available, absolutely love how you kept it simple! I put together a fully functioning pubsub into my containers, set up a survey for health checks, a simple Dockerfile for dynamic configuration of new cluster nodes - nats.io > docs > implementation in under 90 minutes this bad boy just just works!

I have a JSON structure with a text field (which also happens to be JSON in this failure case), and intermittently receive a JSON parse error when parsing that field. I was able to reproduce (sometimes) using the following:

https://gist.github.com/ahallock/38afc33f1f6d16064d5c

Versions:
node-nats 0.4.0
https://hub.docker.com/r/apcera/gnatsd/ (latest)
iojs 1.8.4

When the parsing error occurs, the incoming message appears "garbled" with unknown characters:
��&��I�,��!�f�1�7�|)�:�|))]&��)]

I'll dig into this more as well (nats is awesome!)

Working out an issue implementing the node-nats-streaming library and I was initially confused by the stack trace until I realized that it's because the Error objects are being instantiated as soon as the library was required, causing the stack trace to show the trace to the point the objects are instantiated, not the point that they're thrown.

Obvious suggestion is to set the error vars to strings and instantiate Error when throwing, e.g. throw(new Error(BAD_SUBJECT_ERR)). This change also seems slightly more efficient, as right now you have error objects always hanging around in memory whether they're needed or not.

Before this change, the stack trace looks like this:

<path>/node_modules/node-nats-streaming/node_modules/nats/lib/nats.js:938
      throw(BAD_SUBJECT_ERR);
      ^
Error: Subject must be supplied
    at Object.<anonymous> (<path>/node_modules/node-nats-streaming/node_modules/nats/lib/nats.js:70:23)
    at Module._compile (module.js:409:26)
    at Object.Module._extensions..js (module.js:416:10)
    at Module.load (module.js:343:32)
    at Function.Module._load (module.js:300:12)
    at Module.require (module.js:353:17)
    at require (internal/module.js:12:17)
    at Object.<anonymous> (<path>/node_modules/node-nats-streaming/node_modules/nats/index.js:1:80)
    at Module._compile (module.js:409:26)
    at Object.Module._extensions..js (module.js:416:10)
    at Module.load (module.js:343:32)
    at Function.Module._load (module.js:300:12)
    at Module.require (module.js:353:17)
    at require (internal/module.js:12:17)
    at Object.<anonymous> (<path>/node_modules/node-nats-streaming/lib/stan.js:15:12)
    at Module._compile (module.js:409:26)

After the suggested change:

Error: Subject must be supplied
    at Client.publish (<path>/node_modules/node-nats-streaming/node_modules/nats/lib/nats.js:938:13)
    at Client.request (<path>/node_modules/node-nats-streaming/node_modules/nats/lib/nats.js:1116:8)
    at Stan.subscribe (<path>/node_modules/node-nats-streaming/lib/stan.js:490:11)
[ stack trace from my code truncated ]

As you can see, the second stack trace is informative, the first is not.

Happy to submit a pull request for this unless you want to handle it a different way.

Especially subscriptions that do not have any options could be interesting to use a reference counted sharing for. Would be a nice performance optimisation.

On the other side of the coin. Does the nats server internally share subscriptions to the same client, i.e. if the same client subscribes to the same subject twice I would assume the server will only send matching messages once?

I keep going back to subscribe to remind myself what the signature of the callback should be and I keep missing it...

This is an issue for us. We usually start up all our systems and then have to wait 5-10 minutes for the switches to startup. But when they start up our nats application won't reconnect, it has already failed.

This behaviour should probably be configurable...

It would be useful (especially when wrapping nats) if there was some notification of when subscriptions are automatically ended, e.g. on('unsubscribed', (sid) => console.log(sid))

This wildcard subcription code:

nats.subscribe('*', function (msg, reply, subject) {
    console.log(Date(), '[' + subject + ']:', msg);
});

displays the 'request' message from a nats.request(), but not the 'reply' message.

Is that expected? Is there a way to see both legs of a NATS Request/Reply message transaction from a wildcard subscribe? I hoped to use this as a simple message debug monitor, but need to also see the reply leg to observe all the message traffic.

Thanks for NATS. It's great.

I understand that automatic syncing of clients with servers is planned, in the meantime, is there anything that prevents me from "manually" adding new servers myself once a client have been constructed? I see servers is an exposed property, the client constructor loops through servers opts and pushes them to servers here https://github.com/nats-io/node-nats/blob/master/lib/nats.js#L231

Server on the other hand is a private API https://github.com/nats-io/node-nats/blob/master/lib/nats.js#L252 but it's a simple object which I can either override or expose outside the client.

Alternatively if you expose Server then I can do something like nc.servers.push(new nats.Server(uri)).

Hi,

While debugging a few issues, I found out that SUB is sent more then once since version 0.6.0:

[19686] 2016/04/19 14:03:30.983250 [INF] Starting gnatsd version 0.7.2
[19686] 2016/04/19 14:03:30.984340 [INF] Listening for client connections on 0.0.0.0:4222
[19686] 2016/04/19 14:03:30.984406 [INF] gnatsd is ready
[19686] 2016/04/19 14:03:34.408254 [DBG] 127.0.0.1:53822 - cid:1 - Client connection created
[19686] 2016/04/19 14:03:34.408694 [DBG] 127.0.0.1:53823 - cid:2 - Client connection created
[19686] 2016/04/19 14:03:34.410969 [TRC] 127.0.0.1:53822 - cid:1 - ->> [CONNECT {"lang":"node","version":"0.6.0","verbose":false,"pedantic":true,"name":"service1"}]
[19686] 2016/04/19 14:03:34.411133 [TRC] 127.0.0.1:53822 - cid:1 - ->> [SUB service1.> service1 2]
[19686] 2016/04/19 14:03:34.411160 [TRC] 127.0.0.1:53822 - cid:1 - ->> [PING]
[19686] 2016/04/19 14:03:34.411165 [TRC] 127.0.0.1:53822 - cid:1 - <<- [PONG]
[19686] 2016/04/19 14:03:34.411303 [TRC] 127.0.0.1:53822 - cid:1 - ->> [SUB service1.> service1 2]
[19686] 2016/04/19 14:03:34.411865 [TRC] 127.0.0.1:53823 - cid:2 - ->> [CONNECT {"lang":"node","version":"0.6.0","verbose":false,"pedantic":true,"name":"service2"}]
[19686] 2016/04/19 14:03:34.411887 [TRC] 127.0.0.1:53823 - cid:2 - ->> [SUB service2.> service2 2]
[19686] 2016/04/19 14:03:34.411895 [TRC] 127.0.0.1:53823 - cid:2 - ->> [PING]
[19686] 2016/04/19 14:03:34.411899 [TRC] 127.0.0.1:53823 - cid:2 - <<- [PONG]
[19686] 2016/04/19 14:03:34.411914 [TRC] 127.0.0.1:53823 - cid:2 - ->> [SUB service2.> service2 2]

[19711] 2016/04/19 14:07:10.598530 [INF] Starting gnatsd version 0.7.2
[19711] 2016/04/19 14:07:10.598627 [INF] Listening for client connections on 0.0.0.0:4222
[19711] 2016/04/19 14:07:10.598687 [INF] gnatsd is ready
[19711] 2016/04/19 14:07:16.389572 [DBG] 127.0.0.1:54203 - cid:1 - Client connection created
[19711] 2016/04/19 14:07:16.390074 [DBG] 127.0.0.1:54204 - cid:2 - Client connection created
[19711] 2016/04/19 14:07:16.392446 [TRC] 127.0.0.1:54203 - cid:1 - ->> [CONNECT {"lang":"node","version":"0.5.4","verbose":true,"pedantic":true,"name":"service1"}]
[19711] 2016/04/19 14:07:16.392602 [TRC] 127.0.0.1:54203 - cid:1 - ->> [SUB service1.> service1 2]
[19711] 2016/04/19 14:07:16.392625 [TRC] 127.0.0.1:54203 - cid:1 - ->> [PING]
[19711] 2016/04/19 14:07:16.392629 [TRC] 127.0.0.1:54203 - cid:1 - <<- [PONG]
[19711] 2016/04/19 14:07:16.392933 [TRC] 127.0.0.1:54204 - cid:2 - ->> [CONNECT {"lang":"node","version":"0.5.4","verbose":true,"pedantic":true,"name":"service2"}]
[19711] 2016/04/19 14:07:16.392955 [TRC] 127.0.0.1:54204 - cid:2 - ->> [SUB service2.> service2 2]
[19711] 2016/04/19 14:07:16.392964 [TRC] 127.0.0.1:54204 - cid:2 - ->> [PING]
[19711] 2016/04/19 14:07:16.392967 [TRC] 127.0.0.1:54204 - cid:2 - <<- [PONG]

In 0.4.4 this works fine:

'use strict';

var nats = require('nats'),
    natsCfg = require('./config.json'),
    nc = nats.connect({
       "servers": [
          "nats://user:password@host:port"
       ]
    }
);

nc.subscribe('*', function (msg, reply, subject) {
    console.log(arguments);
});

In 0.5.0 it returns an Authorization Error.

Per PR #57 suggestion.

References PR #55

-Issue reported with how the client parses the opts parameter.
-not entirely intuitive, that a protocol should be required.
-If you pass in something like "consul.service:4222" as the URL, the parseOptions() method will parse that as having a protocol of "consul.service" and lead to a crash
-crash will happen in events.js, which makes it seem like a connection error rather than a string-parsing issue (of course, as we now know, it's both, with each one arising out of our own misunderstanding of the parameter).

The nats server seems to be sending incorrect message size in the message "header". When the client receives a payload in stream.on('data') callback, it uses regular expressions to try and match for the header type (MSG, OK, ERR, PONG, etc) When the message size provided by the server is incorrect, it takes part of the next message and puts it as part of the current message. During the AWAITING_MSG_PAYLOAD step it incorrectly slices payload size so that on the next message inevitably falls through all the regular expression matches in AWAITING_CONTROL step and does nothing.

e.g
Message 1: "MSG my.topic 2 10 {foo:bar}"
Message 2: "MSG my.topic 2 9 {bar:foo}"

Message 1 is interpreted as "MSG my.topic 2 10 {foo:bar}M"
client goes out of sync and does nothing.

This seems like a server issue as well as a client one. The client probably should throw some error when this happens and the server needs to send the correct message size.

We're running gnatsd 5.0 and it's definitely not the latest so I was wondering if there was an issue similar to this that is already fixed.

Our messages are JSON and varies between 15k to 80k in size

Hi,

I can see there is no support for 'name' parameter, it's not accepted when creating a new connection, and not sent to gnatsd.

Is there a plan to add it ?

Support nats streaming in the nats node client.

The execution of resume() implies the check to see if the stream is paused. Thus, wrapping the resume() here is unnecessary.

Moreover, using isPaused() breaks compatibility with Node v0.10.x which, while unarguably old, is required for my use case, presently.

The present scheme of processing callbacks for the publish function is a bit strange. It is roughly akin to: "I have tried to send your message. I don't know whether it was sent or not, but after I threw it away, I was able to ping the server and get a pong back."

I feel that a more appropriate use of the callback is: "I have tried to send your message to the server. The send [failed|succeeded]."

This says nothing about the operation of the server... but then, previous message was only a surrogate indicator, anyway. If the server was unavailable for your message but came up in time to respond to the subsequent 'ping', your app is blithely unaware. It could indirectly be listening for errors or connection losses, but even then, you won't know if that affected the message you sent or not.

Instead, I think the publish method's callback should be very specific in meaning, scoped to only the things it can do something about: decide whether to try again.

This is strictly my opinion, and I'm very interested in hearing alternative views, particularly since I am rather new to NATS.

Right now it is a little ugly to get the sid into the callback if one wants unsubscribe inside of the subscription callback.

It would be great if we could get nats to work over websockets (imho), in theory we just need to shim net, is it ok if I give it a go and send a PR?

The latest versions of TypeScript can pull typescript definitions from an NPM repository. This helps
ensure that the TypeScript definitions are always in sync with what is published in node. Furthermore,
it makes it much easier for TypeScript users to use the node module because all the have to do is
'npm install nats' and they automatically get TypeScript definitions. With Angular2's adoption of TypeScript (and TypeScript's rising popularity, in general), this level of support for TypeScript will make life easier for a lot of developers.

All this means, for the nats developers, is including one file in their package. It doesn't force any of you users to know or use TypeScript. It just provides a definition file that can be found automatically by the TypeScript compiler.

Would you accept a pull request for such a definition file to be redistributed with the nats package in npm?

Is there a reason the code doesn't run on node 10.22 or should the package.json just be updated to allow the most current node runtime environment?

I get the following error on Mac OS X 10.9 running "npm install nats"

[email protected]: wanted: {"node":">= 0.4.x <= 0.10.x"} (current: {"node":"v0.10.22","npm":"1.3.14"})

If I install it manually and run node_pub.js and node_sub.js against gnatsd the daemon report...

panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xa code=0x2 addr=0x0 pc=0xb5a9c]

goroutine 1 [running]:
runtime.panic(0x243d40, 0x578e6d)
/usr/local/go/src/pkg/runtime/panic.c:266 +0x9a
sync/atomic.AddUint64(0x1122f8fc, 0x1, 0x0, 0x0, 0x11241580)
/usr/local/go/src/pkg/sync/atomic/asm_386.s:118 +0xc
github.com/apcera/gnatsd/server.(_client).initClient(0x11249800)
/Users/hans/dev/gosource/src/github.com/apcera/gnatsd/server/client.go:101 +0x56
github.com/apcera/gnatsd/server.(_Server).createClient(0x1122f8c0, 0x66ecd8, 0x11200868, 0x0)
/Users/hans/dev/gosource/src/github.com/apcera/gnatsd/server/server.go:355 +0x8a
github.com/apcera/gnatsd/server.(_Server).AcceptLoop(0x1122f8c0)
/Users/hans/dev/gosource/src/github.com/apcera/gnatsd/server/server.go:296 +0x56c
github.com/apcera/gnatsd/server.(_Server).Start(0x1122f8c0)
/Users/hans/dev/gosource/src/github.com/apcera/gnatsd/server/server.go:197 +0x8a
main.main()
/Users/hans/dev/gosource/src/github.com/apcera/gnatsd/gnatsd.go:89 +0x727

goroutine 5 [chan receive]:
github.com/apcera/gnatsd/server.func·003()
/Users/hans/dev/gosource/src/github.com/apcera/gnatsd/server/server.go:148 +0x36
created by github.com/apcera/gnatsd/server.(*Server).handleSignals
/Users/hans/dev/gosource/src/github.com/apcera/gnatsd/server/server.go:154 +0xe7

goroutine 4 [syscall]:
os/signal.loop()
/usr/local/go/src/pkg/os/signal/signal_unix.go:21 +0x1e
created by os/signal.init·1
/usr/local/go/src/pkg/os/signal/signal_unix.go:27 +0x31

Everything works fine on Windows 7. Damn, did I just say that!

Setting the json option to true makes the msg argument for the publish method non-optional.

const nats = require("nats").connect({ "url": "...", "json": true });

Invoking nats.request("topic", callback) will now cause an error because of the type check of msg https://github.com/nats-io/node-nats/blob/master/lib/nats.js#L990. The msg argument gets defaulted to EMPTY (which is the empty string '').

It seems a bit unintuitive that the signature for the method changes based on an option. I would still like to send empty messages but let the nats library automatically JSON.parse the responses from request invocations.

In ruby_nats, subscribing to "a.b.c.d" will trigger a function when another process publishes "a.b.." (or any wildcard variation of a.b.c.d). It does not appear to behave the same way in node_nats.

nats.subscribe("A.B.C.D", function(msg,reply,subject) {});

then

nats-pub A.B.C.D 'message'
node.js responds

nats-pub A.B.. 'message'
node.js has no response

The inverse behavior, e.g. subscribe "A...*", respond to "A.B.C.D", triggers as expected.

Shouldn't requests subscriptions be auto-unsubscribed, i.e. {max: 1}? As it is now I believe a lot of people are leaving dangling subscriptions when using the request method.

If not I believe this needs to be more explicitly documented. If I hadn't read the code I wouldn't have realised as I don't believe even the examples unsubscribe.

After upgrading to Node 5.x (nats 0.5.4) from 4.x (nats 0.4.4) I'm no longer able to connect (I get a connection timed out). While troubleshooting that, I decided to add event listeners for all connection related events that I saw being emitted from the library. I noticed that only the error events seem to be emitted in a connection timed out scenario. This creates a blind spot when it comes to connection handling.

I noticed the following:

1. There is no connecting event (or retrying) to monitor connection status/progress
2. Since connect calls a constructor under the hood and returns a client synchronously; the included examples lack robust error and connection handling.
3. There is no example of how to configure the timeout, it seems to be connection attempts * connection delays (both look like they're constants).

Here is an example of the code:

nc.on('error', function(error) {
   if (process.env.NODE_ENV !== 'production')  {
       throw error;
   } else {
       console.error('NATS: Error ', error);
   }
});

nc.on('reconnecting', function() {
   console.log('NATS: reconnecting...');
});

nc.on('disconnect', function() {
   console.log('NATS: disconnected');
});

nc.on('connect', function() {
   console.log('NATS: connected');
});

nc.on('reconnect', function() {
   console.log('NATS: Reconnected');
});

Doesn't the protocol specify the casing?