mqttjs / mqtt-packet Goto Github PK

The uint16 cache from numbers.js currently consumes roughly 10Mb, which is too excessive for our application.

We could disable the precache generation, but I think the implementation could be improved.

Currently the cache is an object that maps indexes from 0 to 65535 to 65536 individual Uint8Array objects (using Buffer.allocUnsafe). Each one of these Uint8Array objects is consuming 72 bytes on Chrome (when it's supposed to be representing 2 bytes of unsigned integer data, but each Uint8Array object has quite a bit of overhead).

Another potential issue is we are forcing Big Endian encoding by serializing the bytes manually:

  buffer.writeUInt8(i >> 8, 0)
  buffer.writeUInt8(i & 0x00FF, 0 + 1)

Instead of creating this map of individual arrays, could the same cache be achieved using a single UInt16Array?

I.e.:

const max = 65536
const cache = new Uint16Array(max)

for (let i = 0; i < max; i++) {
  cache[i] = i;
}

With this implementation, the entire cache only consumes roughly 131Kb a significant decrease from 10Mb.

3.4.2.2.1 Property Length
The length of the Properties in the PUBACK packet Variable Header encoded as a Variable Byte Integer. If the Remaining Length is less than 4 there is no Property Length and the value of 0 is used.

Spec says that if remaining len is < 4, there's no property length. I think 'there's no property length' implies that it's not part of the byte stream (beacause it checks out with mosquitto)

const mqtt = require('mqtt-packet');

let puback = {
  cmd: 'puback',
  messageId: 42,
}

const opts = { protocolVersion: 5 }; // default is 4. Usually, opts is a connect packet
let buf = mqtt.generate(puback, opts)
console.log(buf.toString('hex').match(/../g).map((x) => `0x${x}`).join(', '))

Output of the above code is

0x40, 0x03, 0x00, 0x2a, 0x00

Output of ack from mosquitto with no reason code and properties is this

[
    0x40,
    0x2,
    0x0,
    0x9,
]

Is the last 0x00 necessary?

I am trying to connect to an MQTT server over websocket(was actually, but I think it doesn't matter).
But the connection attempt fails, because the client sends the connect packet in little chunks:

I tried to connect with replacing the whole packet generation with a hardcoded one, and it immediately succeeds to connect:

conbuf = Buffer.from([0x10, 0x38, 0x00, 0x04, 0x4d, 0x51, 0x54, 0x54, 0x04, 0xc2, 0x00, 0x1e, 0x00, 0x20, 0x36, 0x37, 0x61, 0x64, 0x33, 0x63, 0x63, 0x35, 0x63, 0x31, 0x31, 0x33, 0x34, 0x35, 0x63, 0x31, 0x39, 0x37, 0x64, 0x38, 0x33, 0x36, 0x31, 0x39, 0x61, 0x63, 0x64, 0x37, 0x30, 0x66, 0x31, 0x63, 0x00, 0x04, 0x74, 0x65, 0x73, 0x74, 0x00, 0x04, 0x74, 0x65, 0x73, 0x74]) stream.write(conbuf)

The same problem appears with the subscribe command.

The Payload Format Indicator should be int8, not byte.

https://docs.oasis-open.org/mqtt/mqtt/v5.0/os/mqtt-v5.0-os.html#_Toc3901111

Here are a couple relevant places that would likely need to change.

The current version in use (4.0.2) has a security vulnerability.

See GHSA-pp7h-53gx-mx7r

Unfortunately our client decided to trust a third company that sold them tablets with a chromium 46 on them. So we need to support that old engine.

When I try to run your lib on the browser the following error appears

Uncaught SyntaxError: Block-scoped declarations (let, const, function, class) not yet supported outside strict mode./node_modules/mqtt-packet/parser.js @ app.js:8170
webpack_require @ app.js:770
fn @ app.js:130
(anonymous function) @ mqtt.js?3409:1
./node_modules/mqtt-packet/mqtt.js @ app.js:8115
webpack_require @ app.js:770
fn @ app.js:130
.......

That file points to your Parser

/*!********************************************!*\
  !*** ./node_modules/mqtt-packet/parser.js ***!
  \********************************************/
/*! no static exports found */
/***/ (function(module, exports, __webpack_require__) {

eval("const bl = __webpack_require__(/*! bl */ \"./node_modules/bl/bl.js\")\nconst EventEmitter = __webpack_require__.........

/***/ }),



const bl = require('bl')
const EventEmitter = require('events')
const Packet = require('./packet')
const constants = require('./constants')
const debug = require('debug')('mqtt-packet:parser')

class Parser extends EventEmitter {

But that's of course not legit on old browsers.
Would it be possible to run your lib through babel and convert it propperly?

The following commit altered parser.js _parseLength such that it generates an error state whenever _parseVarByteNum returns false:

commit 3eb494a19d6a5c855ab098ab6975a934b0bff00a
Author: Hans Klunder <[email protected]>
Date:   Tue Sep 8 11:59:32 2020 +0200

    Fix: restrict Variable Byte Integer to 24bits (#90)

That was to cater for the new max bytes check in _parseVarByteNum, that breaks the case where the remainder of the length header bytes are not available yet (this._list.length <= bytes), since that case is not an error, since it is just waiting for more bytes to be available.

It is relatively rare that it hits this problem, as it only happens if delayed incoming bytes occur at just the right (wrong) time, i.e. when parsing a multi-byte length header field. However, under high traffic we see this occurring a couple times a day since upgrading to mqtt-packet 6.6.0. I have tested winding back to 6.5.0 and we don't hit the problem there.

This is quite serious as when this occurs the byte stream does not get consumed properly, and after the first "Error: Invalid length" error occurs, parsing subsequent incoming bytes gets totally confused and emits numerous errors (losing the data), until the mqtt connection is dropped and re-established.

What do you folks thinks?

http://npm.im/standard

E.g. when connecting to test.mosquitto.org, no reason code is delivered in connack packet

Hi, : )
mqtt-packet looks really neat! ! I'm wondering why is the version different between the client and server ?

server: mosca dependency is mqtt-connection and mqtt-connection dependency is mqtt-packet v3.5.0
client: mqtt -> mqtt-packet v5.6.0

and some functions in package is different

In v5.6 like this

var stream = new Accumulator() // in generate.js
stream.write(buffer) // in writeToStream.js

but in v3.5 maybe

buffer.writeUInt8() // in generate.js that has become writeToStream.js (v5.6.0)

Look forward to your reply.
Thank you! 🍰

ReferenceError: writeNumberCached is not defined
at Object. (/home/travis/build/ioBroker/ioBroker.mqtt/tmp/node_modules/iobroker.mqtt/node_modules/mqtt-connection/node_modules/mqtt-packet/writeToStream.js:14:19)
at Module._compile (module.js:409:26)
at Object.Module._extensions..js (module.js:416:10)
at Module.load (module.js:343:32)
at Function.Module._load (module.js:300:12)
at Module.require (module.js:353:17)
at require (internal/module.js:12:17)
at Object. (/home/travis/build/ioBroker/ioBroker.mqtt/tmp/node_modules/iobroker.mqtt/node_modules/mqtt-connection/node_modules/mqtt-packet/generate.js:4:21)
at Module._compile (module.js:409:26)
at Object.Module._extensions..js (module.js:416:10)

The value of the Protocol Version field for version 5.0 of the protocol is 5 (0x05)

The value of the Protocol Level field for the version 3.1.1 of the protocol is 4 (0x04)

The Server MUST respond to the CONNECT Packet with a CONNACK return code 0x01 (unacceptable protocol level) and then disconnect the Client if the Protocol Level is not supported by the Server [MQTT-3.1.2-2].

The parser emits an error if I feed it a packet with protocol level of 5. This means the mqtt-connection never emits the connect packet, even though it's well formed. It should be the responsibility of the connect handler in the server implementation to validate the protocol level and comply with MQTT-3.1.2-2

I try to connect to my mosquitto broker.
The problem im facing, as soon as the "connect" package is writen, the tcp socket is closed.
I guess this happens because the package is invalid/the broker expect something else?

Code i used for testing:

const net = require("net");

const mqtt = require("mqtt-packet");
const parser = mqtt.parser({
    protocolVersion: 4
});

const socket = net.Socket();

socket.on("close", () => {
    console.log("connection closed");
});

socket.on("data", (data) => {

    console.log(">", data);

    parser.once('packet', packet => {
        console.log(packet)
    });

    parser.parse(data);

});

socket.on("connect", () => {

    console.log("Connected to tcp://open-haus.lan:1883");

    let data = mqtt.generate({
        cmd: 'connect',
        protocolId: 'MQTT', // Or 'MQIsdp' in MQTT 3.1 and 5.0
        protocolVersion: 4, // Or 3 in MQTT 3.1, or 5 in MQTT 5.0
        clean: true, // Can also be false
        clientId: 'my-device-test-node',
        //keepalive: 0, // Seconds which can be any positive number, with 0 as the default setting
        will: {
            topic: '#',
            payload: Buffer.from('dead'), // Payloads are buffers
        }
    });

    socket.write(data, () => {
        console.log("connect has writen", data)
    });

});


socket.connect(1883, "open-haus.lan");

Output:

Connected to tcp://open-haus.lan:1883
connect has writen <Buffer 10 28 00 04 4d 51 54 54 04 06 00 00 00 13 6d 79 2d 64 65 76 69 63 65 2d 74 65 73 74 2d 6e 6f 64 65 00 01 23 00 04 64 65 61 64>
connection closed

The "data" event on the tcp socket isnt even fired.

As seen in this graphic, after the tcp socket is established, a connect packet is send to the broker:

I dont get anything back.

Hi all,

thank you for providing this very helpful package :-).

The MQTT standard version 5 allows clients to connect using a zero-length client identifier. Yet the generate function seems not to be able to handle this:

const mqtt = require('mqtt-packet');
const opts = {protocolVersion: 5};
const parser = mqtt.parser(opts);

parser.on('packet', (packet) => {
    console.log("packet event emitted on noopParser", packet);
});

parser.on('error', (error) => {
    console.log("error event emitted on noopParser", error.message);
});

let packet = {
    cmd: "connect",
    protocolId: "MQTT",
    protocolVersion: 5,
    clean: true,
    keepalive: 60,
    properties:
        {
            receiveMaximum: 20
        },
    clientId:""
};

let generatedBuffer = mqtt.generate(packet, opts);
console.log('generatedBuffer',generatedBuffer);
parser.parse(generatedBuffer);

This will give error message "Packet too short" when parsing the generated buffer.

Thanks in advance for your feedback and best regards

Jannis

In current implementation a numCache is create by default (file numbers.js) and it contains 65536 buffers with predefined values. That leads to quite high memory consumption: ~14Mb of memory (node process) can be saved.

For sure it's always a trade off between memory and performance that's why I'd like to propose to configure numCache: to give an ability to either create cache during start or create buffers on demand and, potentially, cache created.

I did a quick comparison (on benchmarks/writeToStream.js):

• current implementation gives

Total time 2703
Total packets 1000000
Packet/s 369959.3044765076

• a quickly patched version:

var buffer;

const handler = {
    get: function (target, name) {
        const i = ~~name;

        buffer = new Buffer(2);
        buffer.writeUInt8(i >> 8, 0, true);
        buffer.writeUInt8(i & 0x00FF, 0 + 1, true);
        return buffer;
    }
};

module.exports = new Proxy({}, handler);

gives:

Total time 3156
Total packets 1000000
Packet/s 316856.7807351077

Of course this is not a "production"-ready patched code and it can be improved. That's rather an example to show that performance degradation is not so big.

I do understand that most developers prefer speed over memory consumption that's why I propose to leave it as an option.

Please let me know if it works for you. I can prepare a PR with a good solution.

Thanks!

https://github.com/mqttjs/mqtt-packet#disconnect

{
  cmd: 'disconnect'
}

Based on my cursory glance, it looks like specifying a subscription identifier is not supported for the SUBSCRIBE packet. At the very least, it's not part of the ISubscribePacket type definition. I currently don't have time to fix it but I'm posting the issue to document this. I may come back with a PR at a later point.

If someone has the time to investigate this and make a PR, feel free.

In the writeToStream() code there is no use of callbacks or checking the return value on write() calls. For instance here on connect:

As a result the only way to see if the packet has failed to send is to listen for an event on the stream.

In aedes, there is an implicit assumption that whenever writeToStream() is called, the buffer highwatermark will be hit and then once the buffer drains 'drain' will be called. See the aedes write.js file: https://github.com/moscajs/aedes/blob/main/lib/write.js

What is the assumed behavior for writeToStream()? Do we assume that whenever we call it, the highwatermark will be hit and subsequently the drain will be called? Or is that a false assumption being made by the aedes code? What is the best way to check if the call succeeded to write the data to the stream or if it failed?

@mcollina do you have insights into this since you helped write mqtt-packet and the new aedes code?

Hi @mcollina,
Hope you are doing well.

I've been getting this unhandled exception in production randomly:

TypeError: Invalid data, chunk must be a string or buffer, not undefined
    at Socket.write (net.js:704:11)
    at writeNumberCached (/home/ubuntu/app/node_modules/mqtt-packet/writeToStream.js:826:17)
    at publish (/home/ubuntu/app/node_modules/mqtt-packet/writeToStream.js:349:16)
    at generate (/home/ubuntu/app/node_modules/mqtt-packet/writeToStream.js:35:14)
    at Writable.write [as _write] (/home/ubuntu/app/node_modules/mqtt-connection/lib/writeToStre$
    at doWrite (_stream_writable.js:397:12)
    at writeOrBuffer (_stream_writable.js:383:5)
    at Writable.write (_stream_writable.js:290:11)
    at Connection.Duplexify._write (/home/ubuntu/app/node_modules/duplexify/index.js:208:22)
    at doWrite (/home/ubuntu/app/node_modules/readable-stream/lib/_stream_writable.js:428:64)

The line throwing this exception:

It appears the numCache may not have been populated when writeNumberCached was called. Any idea why or how this could happen?

A fix could look like this:

function writeNumberCached (stream, number) {
  return stream.write(numCache[number] || generateNumber(number))
}

I want to redo the writing system as mentioned in the title. That'll give us less overhead by ws/wss connections, generate less payload when processing fewer socket frames. @mcollina, what do you think about it?

Hi,
as part of a production pen test for an application using Aedes as MQTT broker, a question came up how the broker handles invalid UTF-8 code points in topic strings.
According to the MQTT spec (cmp. http://docs.oasis-open.org/mqtt/mqtt/v3.1.1/errata01/os/mqtt-v3.1.1-errata01-os-complete.html#_Toc442180829) and as highlighted by a whitepaper from Trendmicro (https://documents.trendmicro.com/assets/white_papers/wp-the-fragility-of-industrial-IoTs-data-backbone.pdf, section 2.1.2) some code points (i. e. control characters) MUST close the network connection, for some others it MAY close the network connection (e. g. U+0001..U+001F, U+007F..U+009F). However, I could not find any of the filtering anywhere in the code of neither Aedes nor mqtt-packet, which I was supposing to be the relevant candidate for doing so (as it provides the parser for the topic).

The conformance statements from the spec:

The character data in a UTF-8 encoded string MUST be well-formed UTF-8 as defined by the Unicode specification [Unicode] and restated in RFC 3629 [RFC3629]. In particular this data MUST NOT include encodings of code points between U+D800 and U+DFFF. If a Server or Client receives a Control Packet containing ill-formed UTF-8 it MUST close the Network Connection [MQTT-1.5.3-1].

A UTF-8 encoded string MUST NOT include an encoding of the null character U+0000. If a receiver (Server or Client) receives a Control Packet containing U+0000 it MUST close the Network Connection [MQTT-1.5.3-2].

The data SHOULD NOT include encodings of the Unicode [Unicode] code points listed below. If a receiver (Server or Client) receives a Control Packet containing any of them it MAY close the Network Connection:

U+0001..U+001F control characters 
U+007F..U+009F control characters 
Code points defined in the Unicode specification [Unicode] to be non-characters (for example U+0FFFF) 

A UTF-8 encoded sequence 0xEF 0xBB 0xBF is always to be interpreted to mean U+FEFF ("ZERO WIDTH NO-BREAK SPACE") wherever it appears in a string and MUST NOT be skipped over or stripped off by a packet receiver [MQTT-1.5.3-3].

Is this part of the spec just not implemented anywhere or am I looking at the wrong code base?

Caused by trying to decode a V4 connack using code expecting a V5 connack (because V5 has been set in options).

Underlying issue traced to _parseVarByteNum function in parser.js which doesn't check buffer length before reading additional UInt8.

The Server MUST validate that the reserved flag in the CONNECT Control Packet is set to zero and disconnect the Client if it is not zero [MQTT-3.1.2-3]

If I set that flag to 1, mqtt-packet still parses it just fine, but doesn't emit raw flag bits, so it's impossible for a server implementation to comply with this spec requirement using mqtt-packet.

> var parser = require('mqtt-packet').parser()
> parser.on('packet', console.log)
// the `3` in the following sets the reserved flag to 1
> parser.parse(Buffer.from('100c00044d515454040300000000', 'hex'))
Packet {
  cmd: 'connect',
  retain: false,
  qos: 0,
  dup: false,
  length: 12,
  topic: null,
  payload: null,
  protocolId: 'MQTT',
  protocolVersion: 4,
  clean: true,
  keepalive: 0,
  clientId: ''
}

This goes for many other packet types' flags as well...for example

Bits 3,2,1 and 0 of the fixed header in the PUBREL Control Packet are reserved and MUST be set to 0,0,1 and 0 respectively. The Server MUST treat any other value as malformed and close the Network Connection [MQTT-3.6.1-1].

mqtt-packet should either attach a raw buffer to the packet, or at least some fields that represent the raw flag values.

It might be sufficient to emit an error when any packet's flags are invalid; I think in all cases the server is just supposed to close the connection, but I'm not 100% sure.

hi team, I received the following warning on my app build, after tracing it down, I found out mqtt-packet still uses Buffer() here "https://github.com/mqttjs/mqtt-packet/blob/master/constants.js#L131". Can we please get it fixed? thanks,

"(node:8008) [DEP0005] DeprecationWarning: Buffer() is deprecated due to security and usability issues. Please use the Buffer.alloc(), Buffer.allocUnsafe(), or Buffer.from() methods instead."

Would a PR to convert this to Typescript be accepted?
I am trying to refactor MQTT.js, and I am running into trouble because of the massive client options object, which goes from containing pieces of require('url').parse output to being used as the prototype of connect packet.
Having small atomic interfaces for the parameters to connect, publish, etc would be useful.

`mqtt.Client#publish(topic, message, [options], [callback])
Publish a message to a topic

topic is the topic to publish to, String
message is the message to publish, Buffer or String
options is the options to publish with, including:
qos QoS level, Number, default 0
retain retain flag, Boolean, default false
dup mark as duplicate flag, Boolean, default false
properties: MQTT 5.0 properties object
payloadFormatIndicator: Payload is UTF-8 Encoded Character Data or not boolean,
messageExpiryInterval: the lifetime of the Application Message in seconds number,
topicAlias: value that is used to identify the Topic instead of using the Topic Name number,
responseTopic: String which is used as the Topic Name for a response message string,
correlationData: used by the sender of the Request Message to identify which request the Response Message is for when it is received binary,
userProperties: The User Property is allowed to appear multiple times to represent multiple name, value pairs object,
subscriptionIdentifier: representing the identifier of the subscription number,
contentType: String describing the content of the Application Message string
cbStorePut - function (), fired when message is put into outgoingStore if QoS is 1 or 2.
callback - function (err), fired when the QoS handling completes, or at the next tick if QoS 0. An error occurs if client is disconnecting.`

In fact there is no definition for 'properties' in source code

`export interface IClientPublishOptions {
/**

• the QoS
  /
  qos: QoS
  /*
• the retain flag
  /
  retain?: boolean
  /*
• whether or not mark a message as duplicate
  /
  dup?: boolean
  /*
• callback called when message is put into outgoingStore
  */
  cbStorePut?: StorePutCallback
  }`

It seems legit to having DISCONNECT packet of 0xe0 0x00, according to mosquitto, which means the remaining length is 0.

However parser.js:439:Parser.prototype._parseDisconnect() would not check whether the packet.length > 0. Please fix this issue.

As reported in mqttjs/MQTT.js#417, there is an Unexpected identifier error in mqtt-packet. Currently there are two 160 unicodes (Non-breaking space) vs 32 (Normal space) at https://github.com/mqttjs/mqtt-packet/blob/master/parser.js#L355 and https://github.com/mqttjs/mqtt-packet/blob/master/parser.js#L370

In here, IPublishPacket description is this.

Only the topic property is mandatory.

But in /types/index.d.ts, IPublishPacket definition is this.

I think so key of qos, dup, retain, payload (and cmd) is need change to optional.

If this is correct, I can send PR.
Thanks.

the Chrome parser failes on generate.js due to some unexpected characters:
0xc2 0xa0 in front of "" and clean

$ md5sum generate.js
c649514871233fb093c4461f8395e054  generate.js

$ grep -P -n "[\x80-\xFF]" generate.js
44:    , clientId = opts.clientId || ""
76:     (clientId || clean)) {

$ grep  -P "[\x80-\xFF]" generate.js | xxd
0000000: 2020 2020 2c20 636c 6965 6e74 4964 203d      , clientId =
0000010: 206f 7074 732e 636c 6965 6e74 4964 207c   opts.clientId |
0000020: 7cc2 a022 220a 2020 2020 2028 636c 6965  |.."".     (clie
0000030: 6e74 4964 207c 7cc2 a063 6c65 616e 2929  ntId ||..clean))
0000040: 207b 0a                                   {.

Thanks!
peter

Looks like v8.0.0 is not published on NPM yet, could it be pushed out?
EDIT: It's actually on NPM under the 'next' tag. Any reason why it can't put under latest?

have encountered a problem related nextTick.

nextTick is not a function

as per my view its coming from here

and making that line nextTick.nextTick() is working fine.

Hi, I want to create 'puback' package which has 'qos' property is 1 but it always 0:

Hi,

We are developing a browser plugin for previewing websocket traffic, and are looking into your package for parsing MQTT-messages. Your package caught our eye; the implementation looks great, good job.

I couldn't help but notice, however, that you depend on bl and some other node.js stuff. Do you think it could be possible to support the browser as well? If so, it would be a great help for us.

Alternatively - do you know of any other parsing library for MQTT? I looked at MQTT.js, but that is a whole client/server-thing, we just need the parsing logic.

I've implemented this test:

test('SUBSCRIBE packet has no payload', (done) => {
  const subscribe: ISubscribePacket = {
    cmd: 'subscribe',
    messageId: 0,
    subscriptions: [
      {
        topic: 'topic',
        qos: 0,
      },
    ],
  }
  const subscribePacket: Buffer = mqttPacket.generate(subscribe)
  const truncSubscribe = Buffer.alloc(2)
  subscribePacket.copy(truncSubscribe, 0, 0, 2)
  truncSubscribe[1] = 0x00 // No payload, Remaining Length = 0

  const parser = mqttPacket.parser(opts)
  parser.on('error', (error) => {
    expect(error.message).toBe('SUBSCRIBE packet MUST have payload')
    done()
  })
  parser.on('packet', () => {
    throw new Error('SUBSCRIBE packet with no payload is invalid')
  })
  parser.parse(truncSubscribe)
})

I've tampered a SUBSCRIBE packet by removing the payload and forcing the Remaining Length field to zero.

But the specification here: https://docs.oasis-open.org/mqtt/mqtt/v5.0/os/mqtt-v5.0-os.html#_Toc3901030
stated that the SUBSCRIBE packet MUST have a payload.

My tampered packet should be invalid or malformed. Nonetheless the parser is correcting parsing the packet emitting the 'packet' event making this test fail.

Is this ok?

Hi,

I noticed there is missing some important validations on the CONNECT command, particularly with regards to Will and QoS.

According to the v5 spec, section 3.1.2.6:

If the Will Flag is set to 0, then the Will QoS MUST be set to 0 (0x00)
If the Will Flag is set to 1, the value of Will QoS can be 0 (0x00), 1 (0x01), or 2 (0x02). A
value of 3 (0x03) is a Malformed Packet.

However these conditions are not handled, so I am free to set:

var object = {
  cmd: 'connect',
  protocolId: 'MQTT',
  protocolVersion: 5,
  username: 'matteo',
  password: new Buffer('collina'),
  retain: false,
  clean: true,
  will: {
    topic: 'mydevice/status',
    qos: 3
  }
}

And this is accepted just fine, the Flags byte becomes 0xDE which is 11011110, notice byte[3-4] (QoS) = 3... but it gets worse:

var object = {
  cmd: 'connect',
  protocolId: 'MQTT',
  protocolVersion: 5,
  username: 'matteo',
  password: new Buffer('collina'),
  retain: false,
  clean: true,
  will: {
    topic: 'mydevice/status',
    qos: 4
  }
}

Settings qos: 4 actually overwrites the 5th bit in the CONNECT flags. When you inspect the Buffer, you'll see it sets the Flags byte to 0xE6 which is 11100110, notice byte[5] (Retain Flag) = 1, even though we specifically defined retain: false.

Hello everyone,

in https://github.com/mqttjs/mqtt-packet/blob/master/writeToStream.js#L10, there is a test based on the string process.version. To my understanding, this heuristic guesses if one can pass arguments to process.nextTick or if wrapping is necessary, in which case tickShim is used: https://github.com/mqttjs/mqtt-packet/blob/master/writeToStream.js#L12 .

The shim is necessary for versions of io.js up to 1.7.1 (nodejs/node#1077), versions that the heuristic do not match.

I don't know if you have any plan on supporting older versions of io.js, I'm myself stuck with this version until nwjs (https://github.com/nwjs/nw.js) releases 0.13 in the stable branch but hot-fixing this shouldn't take too much effort, though (I can initiate a PR if this helps).

Regards,

Quentin

I have a custom broker using mqtt-connection that crashed today because it received a packet that generated a "wrong subscribe header" error.

I think this is what happened:

• mqtt-packet tried to parse the packet, emitted an error event, then emitted a packet event.
• mqtt-connection forwarded the error event to me and converted the packet event into a subscribe event.
• Since the whole packet wasn't parsed, the subscriptions property was undefined. That's where my broker crashed.

IMO, Parser.parse() should skip emitting the packet event as soon as it encounters an error because it seems strange for me to have to validate the packet that mqtt-packet already determined was invalid.

Since the MQTT spec says connections should be closed on protocol violations, maybe Parser.parse() should stop parsing completely when it encounters an error in case there are more packets in the buffer. This would stop mqtt-connection from continuing to emit packets that shouldn't be processed.

If you agree, I can send a PR but I think this is a backwards incompatible change to the API so would require a major bump.

Currently this package includes a number of files and that could be excluded such as the benchmarks folder and the test.js file. Would it be possible to exclude them using the files whitelist in the package.json (preferable) or with a .npmignore file? This would significantly reduce the file size of this package.

Referencing another issue commented by @mcollina:

Oh! I forgot we were doing that in this module. This problem was fixed long ago in Node.js. We should be calling stream.destroy(err) instead.

Would you like to send a PR?

Originally posted by @mcollina in #124 (comment)

Would be happy to open a PR for this, but had a couple of questions here:

1. Wouldn't this be a breaking change?
2. Is this truly the desired behavior? Suppose someone tries to push a CONNECT packet with an invalid protocol version. This is an error, but do we want to destroy the entire stream? Or just fail that one packet?

QoS 0 in puback in https://github.com/mqttjs/mqtt-packet/blob/master/test.js#L1072 ?
It seems that packet is generated from a template https://github.com/mqttjs/mqtt-packet/blob/master/packet.js
and it will appear in the packet variable in https://github.com/mqttjs/mqtt-connection/blob/master/test/connection.parse.js#L365

A little bit confusion

I think the variable name and error message should consistent with the specification as follow.
http://docs.oasis-open.org/mqtt/mqtt/v3.1.1/os/mqtt-v3.1.1-os.html#_Toc398718025

i've got different date type after authorizePublish

After digging a little bit on the library, found what it seems a bug, or at least a miss-documentation. It has mainly affectations on 5.0.

const client = mqttCon(stream, {
    protocolVersion: 5,
    protocolId: 'MQTT',
});
console.log(client.options);

Outputs undefined.

const client = mqttCon(stream);
client.setOptions({
    protocolVersion: 5,
    protocolId: 'MQTT',
});
console.log(client.options);

Outputs { protocolVersion: 5, protocolId: 'MQTT' }.

const client = mqttCon(stream);
client.setOptions({
    protocolVersion: 5,
    protocolId: 'MQTT',
});
client.on('connect', () => {
    console.log(client.options);
});

Outputs:

Packet {
  cmd: 'connect',
  retain: false,
  qos: 0,
  ...

const client = mqttCon(stream);
client.setOptions({
    protocolVersion: 5,
    protocolId: 'MQTT',
});
client.on('connack', () => {
    console.log(client.options);
});

Outputs:

Packet {
  cmd: 'connack',
  retain: false,
  qos: 0,
  ...

This has a big implication, as it causes errors on subscriptions (cannot subscribe as topic cannot be decoded) and also on publish events receives payloads with extra 00 byte at the begining, corresponding to undecoded properties:

Server is sending the content with the space for properties but client is not consuming them, so there are some bytes not being consumed and being left in payload.

Also on the contrary, the client sends a packet with properties bytes codified, and then server is not consuming the properties so they remain in payload, making payload to be displaced so cannot be correctly decoded:

from parser.js:

  // Properties mqtt 5
  if (this.settings.protocolVersion === 5) {
    var properties = this._parseProperties()
    if (Object.getOwnPropertyNames(properties).length) {
      packet.properties = properties
    }
  }

The other point is that options are being overridden by connectPacket in connection.js

Problem:
I tried to use this package with HIVE-MQ-WebSockets client and it does not work. The problem is, that this client (HIVE-MQ) does not expects chunked packets. I understand, that this is a problem of HIVE-MQ, but just now it is my problem :)

Solution:
Build first the whole packet as a buffer and send all bytes together.

https://github.com/GermanBluefox/mqtt-packet/blob/master/writeToStream.js

Result:
After I modified writeToStream.js file and I could communicate with HIVE-MQ and everything was fine. I could not test client part of mqtt-packet and tested only connack, publish, confirmation, suback, emptyPacket.
connect, subscribe and unsubscribe are untested.
I understand, that the performance will decrease slightly, but it is much better, that not working communication.

Please check the changes and say your opinion.

P.S. Just cant find the newest branch and do not understand where to place my pull request.
I see there is a brunch "packet" and the solution there is similar.

Will cause `RangeError [ERR_BUFFER_OUT_OF_BOUNDS]: Attempt to write outside buffer bounds in special packets

https://gist.github.com/blue-bird1/5f79ca8665c4fc9dffb7cd3ed3233490

ERROR in ./node_modules/mqtt-packet/node_modules/bl/BufferList.js
Module build failed (from ./node_modules/babel-loader/lib/index.js):
SyntaxError: "copy" is read-only

78 | BufferList.prototype.copy = function copy (dst, dstStart, srcStart, srcEnd) {
| ^
79 | if (typeof srcStart !== 'number' || srcStart < 0) {
80 | srcStart = 0
81 | }

This issue was initially raised at mqttjs/MQTT.js#1294 but the error seems to point to mqtt-packet. I wil synchronize both if there is a solution

When importing MQTT.js to a VueJS/Quasar app, I get the following error:

app.js:216 Uncaught ReferenceError: Buffer is not defined
    at eval (constants.js?885d:46)
    at Object../node_modules/mqtt-packet/constants.js (vendor.js:1729)
    at __webpack_require__ (app.js:213)
    at fn (app.js:477)
    at eval (parser.js?7aac:4)
    at Object../node_modules/mqtt-packet/parser.js (vendor.js:1779)
    at __webpack_require__ (app.js:213)
    at fn (app.js:477)
    at eval (mqtt.js?e7bd:1)
    at Object../node_modules/mqtt-packet/mqtt.js (vendor.js:1749)
eval @ constants.js?885d:46
./node_modules/mqtt-packet/constants.js @ vendor.js:1729
__webpack_require__ @ app.js:213
fn @ app.js:477
eval @ parser.js?7aac:4
./node_modules/mqtt-packet/parser.js @ vendor.js:1779
__webpack_require__ @ app.js:213
fn @ app.js:477
eval @ mqtt.js?e7bd:1
./node_modules/mqtt-packet/mqtt.js @ vendor.js:1749
__webpack_require__ @ app.js:213
fn @ app.js:477
eval @ client.js?9905:8
./node_modules/mqtt/lib/client.js @ vendor.js:1800
__webpack_require__ @ app.js:213
fn @ app.js:477
eval @ index.js?4c91:3
./node_modules/mqtt/lib/connect/index.js @ vendor.js:1822
__webpack_require__ @ app.js:213
fn @ app.js:477
eval @ index.js??clonedRuleSet-3.use[0]!./node_modules/@quasar/app/lib/webpack/loader.vue.auto-import-quasar.js??ruleSet[0].use[0]!./node_modules/vue-loader/dist/index.js??ruleSet[0].use[1]!./src/App.vue?vue&type=script&lang=ts:6
./node_modules/@quasar/app/lib/webpack/loader.js.transform-quasar-imports.js!./node_modules/ts-loader/index.js??clonedRuleSet-3.use[0]!./node_modules/@quasar/app/lib/webpack/loader.vue.auto-import-quasar.js??ruleSet[0].use[0]!./node_modules/vue-loader/dist/index.js??ruleSet[0].use[1]!./src/App.vue?vue&type=script&lang=ts @ app.js:63
__webpack_require__ @ app.js:213
fn @ app.js:477
eval @ App.vue?vue&type=script&lang=ts:5
./src/App.vue?vue&type=script&lang=ts @ app.js:134
__webpack_require__ @ app.js:213
fn @ app.js:477
eval @ App.vue:6
./src/App.vue @ app.js:96
__webpack_require__ @ app.js:213
fn @ app.js:477
eval @ app.js:6
./.quasar/app.js @ app.js:19
__webpack_require__ @ app.js:213
fn @ app.js:477
eval @ client-entry.js:11
./.quasar/client-entry.js @ app.js:30
__webpack_require__ @ app.js:213
(anonymous) @ app.js:1387
__webpack_require__.O @ app.js:260
(anonymous) @ app.js:1388
(anonymous) @ app.js:1390

I tried (more or less randomly, after finding a similar issue) to

npm install Buffer

which chnaged the error to

Uncaught TypeError: Buffer.alloc is not a function
    at eval (constants.js?885d:128)
    at Array.map (<anonymous>)
    at eval (constants.js?885d:127)
    at Array.map (<anonymous>)
    at eval (constants.js?885d:126)
    at Array.map (<anonymous>)
    at genHeader (constants.js?885d:125)
    at eval (constants.js?885d:140)
    at Object../node_modules/mqtt-packet/constants.js (vendor.js:1707)
    at __webpack_require__ (app.js:213)

Please let me know if I can help with the debug.

Using TypeScript, reading user properties requires something like this (MQTT client):

client.on('message', function(topic, message, packet) {
  const pckt: IPublishPacket = Object.assign(packet);
  if (pckt.properties && pckt.properties.userProperties) {
    const userProperties: {[index: string]: string} = Object.assign(pckt.properties.userProperties);
    console.log(userProperties['test']);
  }
});

It would be better if the userProperties were changed from being {object} to being {[index: string]: string}.

It would also be better if the on message callback (in MQTT) required an IPublishPacket rather than the more generic Packet.

Changing both these would allow type assignments to be removed and the TypeScript code could be changed to:

client.on('message', function(topic, message, packet) {
  if (packet.properties && packet.properties.userProperties) {
    console.log(packet.properties.userProperties['test']);
  }
});

Linked to mqttjs/MQTT.js#1248