bschwind / mqtt-broker

A tokio-based MQTT v5 broker written in pure Rust [WIP]

mqtt-broker

A tokio-based MQTT v5 broker written in Rust.

Project Goals

The goals for this project are fairly straightforward:

• Adhere to the MQTT V5 Spec
• Be easily deployable as a single binary
• Have reasonable performance and memory usage on a single node

I originally started this project as a simple, open-source broker for any IoT products I might make. If I were to sell such products, I would want to allow users to run their own broker in case I can no longer run one, and it should be as easy as possible to do so.

Extra features like a broker cluster or extra transport layers are nice to have features, but won't be considered until the core V5 spec is implemented. The exception to this is the WebSocket transport, which is specifically mentioned in the spec and quite useful to have.

Comparison to Other Brokers

Wikipedia has a fairly comprehensive list of brokers to choose from.

rumqtt at the moment appears to be the most fully-featured broker. Take a look there first if you're looking for a more "ready-to-go" Rust broker.

PubSubRT is another interesting Rust-based broker. It's an alternative to MQTT, not an implementation of it.

NATS.rs seems really nice too, but I haven't looked further into it yet.

Spec Compliance

This broker is currently not compliant with the MQTT V5 spec. Visit the spec compliance milestone to see the current progress.

Dependencies

• cargo
• rustc (version 1.39 or later)

Build

$ cargo build --release

Run

$ cargo run --release

Testing

$ cargo test

Code Format

The formatting options currently use nightly-only options.

$ cargo +nightly fmt

Code Linting

$ cargo clippy

Code Fuzzing

Fuzzing requires a nightly toolchain. Fuzzing for this project is currently confirmed to work with:

rustc 1.42.0-nightly (6d3f4e0aa 2020-01-25)

Running

cargo install cargo-fuzz
cargo +nightly fuzz run decoder_fuzzer_v311
cargo +nightly fuzz run decoder_fuzzer_v500
cargo +nightly fuzz run topic_filter_fuzzer
cargo +nightly fuzz run topic_fuzzer

mqtt-broker's Issues

Implement User Properties Forwarding

Spec: https://docs.oasis-open.org/mqtt/mqtt/v5.0/os/mqtt-v5.0-os.html#_Toc3901116

Implement Receive Maximum

Spec: https://docs.oasis-open.org/mqtt/mqtt/v5.0/os/mqtt-v5.0-os.html#_Flow_Control

Add fuzz targets for decoding

Reference: https://fitzgeraldnick.com/2020/01/16/better-support-for-fuzzing-structured-inputs-in-rust.html

The subscriber does not receive a retain message

I send a message with the retain flag and after that, I run the subscriber. I expect that the subscriber receives the retain message.

Implement Enhanced Authentication

(this is a big one)

Spec: https://docs.oasis-open.org/mqtt/mqtt/v5.0/os/mqtt-v5.0-os.html#_Enhanced_authentication

Support QoS 2

Here's a good resource explaining the various QoS levels:

https://mntolia.com/mqtt-qos-levels-explained/

Log crate instead of println

We're using this broker in a proof of concept and would like to replace the println! logs with the log crate macros. I can push a PR but want tot clarify first if this is something mergeable. I would replace and the println! calls with debug!, info!... and initialise env_logger(?) in the main.

@bschwind fine for you?

Question: Client channel handling with `mpsc::Sender<..>::try_send`

Hello,

by reading the broker code I stumbled across all those try_send calls when the broker sends a packet to the client tasks. Almost everywhere the result is ignored which means that if the channel is full (e.g slow client or high load) the packet is just dropped. The channels to the clients are also configured with a buffer of 5 which immediately leads to packet loss when a client connects and sends with a higher in flight setting.

What is the desired approach here?

I did a experiment and replaced the try_send with a send(..).await in order to perform some throughput tests. This backpressures between everything but is fine for my scenario (closed system with a limited number of well known clients).

Multiplication overflow

Caught by the fuzz tests:

mqtt-broker/mqtt-v5/src/decoder.rs

Lines 103 to 105 in 9037df2

	// TODO(bschwind) - Fuzzer panicked at 'attempt to multiply with overflow'
	// Test with this input: [81, 251, 230, 255, 255, 255]
	value += ((encoded_byte & 0b0111_1111) as u32) * multiplier;

Make matching_subscribers function return an iterator

Assuming we can do it in an efficient way, it would be nicer to have the matching_subscribers function return an iterator instead of accepting an element callback.

Relevant code:

mqtt-broker/mqtt-v5-broker/src/tree.rs

Lines 202 to 241 in 1533c5c

	fn matching_subscribers<'a, F: FnMut(&T)>(&'a self, topic: &Topic, mut sub_fn: F) {
	let mut tree_stack = vec![];
	let levels: Vec<TopicLevel> = topic.levels().collect();
	tree_stack.push((self, 0));
	while !tree_stack.is_empty() {
	let (current_tree, current_level) = tree_stack.pop().unwrap();
	let level = &levels[current_level];
	for (_, subscriber) in &current_tree.multi_level_wildcards {
	sub_fn(subscriber);
	}
	if let Some(sub_tree) = &current_tree.single_level_wildcards {
	if current_level + 1 < levels.len() {
	tree_stack.push((sub_tree, current_level + 1));
	} else {
	for (_, subscriber) in &sub_tree.subscribers {
	sub_fn(subscriber);
	}
	}
	}
	if let TopicLevel::Concrete(level) = level {
	if current_tree.concrete_topic_levels.contains_key(*level) {
	let sub_tree = current_tree.concrete_topic_levels.get(*level).unwrap();
	if current_level + 1 < levels.len() {
	let sub_tree = current_tree.concrete_topic_levels.get(*level).unwrap();
	tree_stack.push((sub_tree, current_level + 1));
	} else {
	for (_, subscriber) in &sub_tree.subscribers {
	sub_fn(subscriber);
	}
	}
	}
	}
	}
	}

Doc Request: Feature Comparison Chart with mosquitto

I think mosquitto is one of the more well-known open source MQTT broker implementations and I think it'd make sense to have a section of the README devoted to a comparison of features. Obviously mosquitto is a very mature project at this point, but I think it'd be helpful as a starting point for discussing new features.

My team is currently using mosquitto in production and have had a couple issues with features breaking in patch builds. While we're not opposed to making bug fixes in mosquitto, we primarily develop in Rust, not C, and it'd be much easier for us to contribute changes or features to a Rust-based broker.

Some features I know we'd be looking for: bridging between brokers, websockets connections, and persistent storage. This issue isn't asking for those things, just for a list of features that are different so I know where I (and others) could start contributing.

Implement Retained Messages

Spec: https://docs.oasis-open.org/mqtt/mqtt/v5.0/os/mqtt-v5.0-os.html#_Toc3901104

Add TLS Support

Ideally with integration for LetsEncrypt to have automatic, self-renewing certificates.

Setup travis CI

Remove usage of unsafe

mqtt-broker/src/topic/tree.rs

Line 174 in 0b684bf

let mut tree = unsafe { &mut *stack_val };

Separate broker and main

The mqtt-v5-broker crate is a binary. The main opens the listeners and forwards clients to the broker state upon connects.
When playing with the broker the code in main is kind of integration specific and one would like to implement specific things prior to the broker start and where to listen etc....

Splitting the crate into a lib and a example main would allow to implement a custom main with custom logging, listeners etc...

Steps are:

• create src/lib.rs
• move the mod from main.rs to lib.rs
• decide what need to be pub exported in lib.rs e.g pub use mqtt-v5-broker::broker::Broker;
• move client_handler, upgrade_stream and websocket_client_handler to client.rs and slightly refactor?

Implement Response Topic

Spec: https://docs.oasis-open.org/mqtt/mqtt/v5.0/os/mqtt-v5.0-os.html#_Toc3901066

Implement Maximum Packet Size

Spec: https://docs.oasis-open.org/mqtt/mqtt/v5.0/os/mqtt-v5.0-os.html#_Toc3901050

Use Bytes instead of Vec<u8> for packet payloads

I think this will reduce data copies but I need to double-check to be sure.

decode_variable_int func bug

According to [1.5.5 Variable Byte Integer] of the MQTT5.0 specification, the algorithm to decode a variable byte integer type is as follows

Algorithm to decode a variable byte integer type

multiplier = 1

value = 0

do

   encodedByte = 'next byte from stream'

   value += (encodedByte AND 127) * multiplier

   if (multiplier > 128*128*128)

      throw Error(Malformed Variable Byte Integer)

   multiplier *= 128

while ((encodedByte AND 128) != 0)

Original

However, the implementation of decoding variable-byte integer types is wrong.
Specifically, the implementation of the arithmetic part of "multiplier" is different.

fn decode_variable_int(bytes: &mut Cursor<&mut BytesMut>) -> Result<Option<u32>, DecodeError> {
    let mut multiplier = 1;
    let mut value: u32 = 0;

    loop {
        let encoded_byte = read_u8!(bytes);

        value += ((encoded_byte & 0b0111_1111) as u32) * multiplier;
        multiplier *= 128;

        if multiplier > (128 * 128 * 128) {
            return Err(DecodeError::InvalidRemainingLength);
        }

        if encoded_byte & 0b1000_0000 == 0b0000_0000 {
            break;
        }
    }

    Ok(Some(value))
}

With the current implementation, if the client sends a PUBLISH packet with a variable-byte integer size of 2,097,152 (0x80,0x80,0x80,0x01) to 268,435,455 (0xFF,0xFF,0xFF,0xFF,0x7F), the following error will be output.

Error while reading frame: InvalidRemainingLength

Listening on 0.0.0.0:1883
Listening on 0.0.0.0:8080
Got a new socket from addr: V4(127.0.0.1:52705)
Handling a client
got a packet: Some(Ok(Connect(ConnectPacket { protocol_name: "MQTT", protocol_version: V500, clean_start: true, keep_alive: 60, session_expiry_interval: None, receive_maximum: Some(ReceiveMaximum(20)), maximum_packet_size: None, topic_alias_maximum: None, request_response_information: None, request_problem_information: None, user_properties: [], authentication_method: None, authentication_data: None, client_id: "", will: None, user_name: None, password: None })))
Client ID s5nsSQSgkIsv-BOYwKdUZ connected (Version: V500)
Error while reading frame: InvalidRemainingLength
rx
Client ID s5nsSQSgkIsv-BOYwKdUZ disconnected

Fixed

The revision is like this.

fn decode_variable_int(bytes: &mut Cursor<&mut BytesMut>) -> Result<Option<u32>, DecodeError> {
    let mut multiplier = 1;
    let mut value: u32 = 0;

    loop {
        let encoded_byte = read_u8!(bytes);

        value += ((encoded_byte & 0b0111_1111) as u32) * multiplier;

        if multiplier > (128 * 128 * 128) {
            return Err(DecodeError::InvalidRemainingLength);
        }

        multiplier *= 128; // here

        if encoded_byte & 0b1000_0000 == 0b0000_0000 {
            break;
        }
    }

    Ok(Some(value))
}

Please let me know if I'm saying something wrong.

Add Travis CI integration

Or perhaps use github actions.

Add constructors for Packet types

Consider implementing Default as well, right now constructing packets is very verbose.

Port spec compliance testing to Rust

Currently, the eclipse paho project has some python-based compliance testing scripts:

V3.11 - https://github.com/eclipse/paho.mqtt.testing/blob/master/interoperability/client_test.py
V5 - https://github.com/eclipse/paho.mqtt.testing/blob/master/interoperability/client_test5.py

It would be nice to port these to Rust, either using this library as a client, or using another open-source and compatibly-licensed Rust project.

Implement Persistent Storage

I think this is not technically part of the spec, but it's very much a desired feature. Given that this is intended to be a simple-to-operate server, I would likely start with either a custom single-file-based storage approach (with a serde-serialized struct containing all desired state), or an sqlite database.

If this broker ever grows into a multi-node broker cluster, we can start thinking about storage solutions which are more appropriate for distributed systems.

Implement Message Expiry Interval

Spec: https://docs.oasis-open.org/mqtt/mqtt/v5.0/os/mqtt-v5.0-os.html#_Toc3901112

Make encoder/decoder return the number of bytes written/read

Implement Topic Alias

Spec: https://docs.oasis-open.org/mqtt/mqtt/v5.0/os/mqtt-v5.0-os.html#_Toc3901113

Implement Shared Groups

Spec: https://docs.oasis-open.org/mqtt/mqtt/v5.0/os/mqtt-v5.0-os.html#_Toc3901250

Implement Subscription Identifiers

Spec: https://docs.oasis-open.org/mqtt/mqtt/v5.0/os/mqtt-v5.0-os.html#_Subscription_Options

Respond to WebSocket pings

Right now the websocket stream is converted to a stream of MQTT packets while ignoring all other WebSocket opcodes. Those extra opcodes should probably be propagated up the stream or have some other mechanism which is able to send pong responses for each WebSocket ping.