hivemq / mqtt-cli

MQTT CLI is a useful command line interface for connecting various MQTT clients supporting MQTT 5.0 and 3.1.1

mqtt-cli is a feature-rich MQTT Command Line Interface.

Getting Started • Publish • Subscribe • Shell • Test • How to Contribute

Getting Started

• Installation instructions
• Building from source
• Full documentation

Publish

• Publish a message
• Quick start: mqtt pub -t your-topic -m "your message" -h your-mqtt-broker.com
• Further documentation

Subscribe

• Subscribe to topics and receive output directly on the console
• Quick start: mqtt sub -t your-topic -h your-mqtt-broker.com
• Further documentation

Shell

• Enter the mqtt-cli shell mode to access more MQTT functionality
• Quick start: mqtt sh
• Further documentation

Test

• Run tests against a broker to find out its features and limitations
• Quick start: mqtt test -h your-mqtt-broker.com
• Further documentation

How to Contribute

• If you want to request a feature or report a bug, please create a GitHub Issue using one of the provided templates
• If you want to make a contribution to the project, please have a look at the contribution guide

mqtt-cli's Issues

Make repo community ready

• Maven to gradle
• add develop as default branch
• Licensing - Apache 2.0
• Adding a Contribution description for the project
• Clean up the project directory
• Add CodeOwners
• Code of Conduct
• ISSUE_TEMPLATES for bug and features u.a.
• Refactor README to add to an external documentation
• CI - Travis adding

List all active subscriptions of a client

In Shell-Mode it might be useful to list the active subscriptions of the currently active context client.

Example:
client@localhost> sub -t topic1
client@localhost> sub -t topic2
client@localhost> ls_subs
topic1
topic2

Is this a good idea or unneeded?

Implement retained message Publish

Add ability to set flag if a Message should be published as retain

@CommandLine.Option(names = {"-r", "--retain"}, required = false, description = "The message is retain.")

Add command to list all clients - when working in shell mode

In shell mode it is possible to connect a couple of mqtt clients.
It is useful to list all clients (available form the cache) with client information
define subcommand ls
mqtt>ls
-> list all clients with client identificator

mqtt>ls -t
-> list all clients with client identificator ordered by creation time

mqtt>ls -a
-> list all clients with client identificator with detailed client information

mqtt>ls -at / or ta
-> list all clients with client identificator ordered by creation time and detailed client information

Test usage of cache

In Shell Mode a cache is used to have the still connected clients.

Test usage of cache and right access also with more than one subscribers, publishers are connected.
Test if a client first connected and later added subscriptions is correct stored in the cache.

• con -i abc
• sub -i abc -t test

Test if Client can be still used in the right ways
Test if cache is cleared after disconnect

Add possibility to define output file for output of a client

Per default all output from subscribing occurs in the console /shell
(if a client gets messages from publishers)
Add possibility
to use as output file named a .log, so that all will be written into the file.
The setting can be used at connect and subscribe

• con -of (file)
• sub -oc (console)

Implement Publish Properties

The publish command needs the following properties added:

-pe --expiryInterval
-pf --payloadFormatIndicator
-pc --contentType
-prt --responseTopic
-pcd --correlationData
-pu --userProperties

Naming should be consistent with the willProperties defined in #4

Implement missing Attributes for Will Message in CONNECT

The Will Message in the Connect Packet need the additional missing Parameters:

willPublish.messageExpiryInterval
willPublish.delayInterval
willPublish.payloadFormatIndicator
willPublish.contentType
willPublish.responseTopic
willPublish.correlationData
willPublish.userProperties

Add - this to the parameter, online help and implementation of the WillMessage

MQTT Implementation example:

https://hivemq.github.io/hivemq-mqtt-client/docs/mqtt_operations/connect.html#will

Complete Implementation of specifying the version

Description

A Client can be run in MQTT 3 or MQTT 5 Mode, - depending on this a couple of features can be used or not.

Default version is V5

implement building of MQTT Client depending on the version

add check and error handling if the used properties from the command line fits to the version

we should decide if we ignore un-useful settings or throw an error in each case

Unify userProperties and tls-version separation

Problem or use case

1. Currently, if more than one user property must be specified they must be separated with a "|" (pipe) symbol.

2. If more than one tls-version to support must be specified it must be separated with a ":" (colon) symbol.

The different separators may result in a confused user and is not an expected behavior.

Preferred solution or suggestions

Use the same separator for both options or omit the separation principle completely and instead require the user to specify the option more than once,
e. g:

hivemq-cli pub -t topic -m msg -up key1=value1 -up key2=value2 -up key3=value3

CLI should be available dmg, homebrew, exe, zip rpm (and deb)

CLI should be available via Package Manager like apt-get, brew, deb, RPM, Launchpad ... to ease the installation process.
Package should include a lightweight JVM

Implement missing Mqtt5 options

In Connect:
userProperties

In Subscribe:
userProperties

In Disconnect:
reasonString
sessionExpiry
userProperties

Implement user properties for CONNECT

Additional to the basic CONNECT - the user should be able to add user properties to the connect package via CLI.

(only if version is set to v5 - default)

implement parameters, with defaults and online help, implementation and test

see https://hivemq.github.io/hivemq-mqtt-client/docs/mqtt_operations/connect.html#user-properties

Automatically generated client identifier consist of too many byte

In the Mqtt specification it is stated that a broker "MUST allow ClientID’s which are between 1 and 23 UTF-8 encoded bytes".

Also "The Server MAY allow ClientID’s that contain more than 23 encoded bytes"

To support any broker the client identifier should always be generated so that it bytes do not exceed 23 in length.

Currently the cli generates a 54-byte string, e. g:
hmqClientMQTT_5_0-74eca7ba-5bfe-410a-a724-b956011ef66f

The createIdentifier method in ConnectCommand should be changed so that a 23-byte string is created as an unique identifier.

Also the broker must only accept the following characters in the identfier "0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ". Therefore also the creation has to be changed so that only these characters are used.

http://docs.oasis-open.org/mqtt/mqtt/v5.0/cs02/mqtt-v5.0-cs02.html#_Toc514345331

Add release package for .nuget

A gradle plugin for .nuget can be found here:
https://github.com/Itiviti/gradle-nuget-plugin

Add the package in the same way like the others
Add documentation

buildscript {
    repositories {
      mavenCentral()
    }

    dependencies {
        classpath "com.ullink.gradle:gradle-nuget-plugin:2.17"
    }
}

apply plugin: 'nuget'

Clients must be identified with all key attributes instead of only the identifier

Input:
hmq> con -i myClient -h mybroker.com -p 1884 -v 3
hmq> pub -i myClient -t test -m msg

Expected:
The publish command uses the previously connected client which is uniquely identified with "myClient".

Result:
Publish connects a new client to default settings.

Reason:
The client lookup in the client cache uses a key consisting of mqtt-version, host, port and identifier instead of using solely the identifier.

Implement ability to read parameter from yaml files

Add ability to read all information from yaml file(s) - this could be a single parameter or a block of parameters or at least the whole command could be packed into a yaml file and read from there:

Identifier for reading from File is @
examples:
mqtt con -u @myname.yaml
mqtt @myFile.yaml

Content should look like described in the example:

mqtt:
_version: 5
method: con
client:
identifier: C1
user: Max
password: "$"
cleanStart: 1
sessionExpiry: 0
will:
properties:
p:
- "p1:x1"
- "p2:x2"
topic: "/myLastWill"
message: "@/path2my/will.yaml"

Implement CONNECT Restrictions

Implement for connect message the ability to restrict with MQTTRestrictions

• add param, online help, implementation and test

see https://hivemq.github.io/hivemq-mqtt-client/docs/mqtt_operations/connect.html#restrictions

Mqtt5ConnectRestrictions restrictions = Mqtt5ConnectRestrictions.builder()
.receiveMaximum(16)
.sendMaximum(32)
.maximumPacketSize(2048)
.sendMaximumPacketSize(1024)
.topicAliasMaximum(16)
.sendTopicAliasMaximum(8)
.requestProblemInformation(false)
.requestResponseInformation(true)
.build();

Mqtt5Connect connectMessage = Mqtt5ConnectMessage.builder()
.restrictions(restrictions)
...
.build();

Beautify console formatting for help messages

The output of --help messages or a wrongly typed command are not very useful if an error occurred because the output is to big.

As a user I would like to see only the error of my wrongly typed command.

Cancelling the subscribe stay option in shell mode will cancel the shell

Behavior:
client@localhost> sub -t test -s
... -> Ctrl + C
^C
The shell is closed.

Expected behavior:
The shell falls back into the context of the client.
client@localhost> sub -t test -s
... -> Ctrl + C
^C
client@localhost>

Add ability to Ping as a client in shell

In Shell-Mode add the possibility after a client was connected to send a ping request to the broker.

e.g:
myClient1> ping

PUBACK has to be awaited after Publish

Input:
pub -i client -t test -m msg -d

Result:
Client client: sending CONNECT
Client client: received CONNACK SUCCESS
Client client: sending PUBLISH: (Topic: test, QoS AT_MOST_ONCE,Message: 'msg')

Expected:
Also
Client client: received Publish Result

Implement missing Connect Message Attributes

Implement the missing basic connect message Attributes

CommandLine.Option -user defaultValue = ""

description - The username for the client UTF-8 String.

CommandLine.Option -password defaultValue = ""

The password for the client UTF-8 String

CommandLine.Option -password defaultValue = "60"

description - A keep alive of the client (in seconds)

CommandLine.Option -cleanStart defaultValue = "true"

description - Define a clean start for the connection.

CommandLine.Option -sessionExpiryInterval defaultValue = "0"

description - Session expiry can be disabled by setting it to 4294967295.

Proguard the jarfile

To optimize the cli the jar file should be proguarded before it is deployed as a package.

https://www.guardsquare.com/en/products/proguard

Refactor Client Cache to a Map?

In the Code Review of #41 @SgtSilvio noted that the use case of the client cache is unclear.

@ahelmbr could you explain why there is a need for a cache implementation and a simple map from identifier to cache is not used?

In Shell-Mode: Use the connection as active context per default and add ability to switch contexts

After connection in the shell mode the user should be act in the current context per default

Input
sh>con -h myHost.com -p 1884
Result

• client is connected and the prompt is working with that client as the active context:
  sh-ClientID@host>

The user should be able to switch between contextes and leave a context (without disconnect)
Input
sh-ClientID@host> switch_context -i 'other-id' -h 'other host'
sh-ClientID2@host2>

Input
sh-ClientID@host> leave_context
sh>

If working in a specific context

• the pub and sub command should not be extended from connect and use always the connection of the active context.
  (naming pub / sub should be the same )
• during active context another conn command should not possible

-s option of subscribe with context prints all active subscriptions of a client

Expected behavior

sub -t topic -s should only print the messages received on topic topic.

Actual behavior

The command prints every publish received on any active subscription of the client.

To Reproduce

Subscriber

mqtt> con 
mqttClient-MQTT_5_0-5ec78216-632c-4bf0-81e0-9a864d42d6cd@localhost> sub -t /topic1/
mqttClient-MQTT_5_0-5ec78216-632c-4bf0-81e0-9a864d42d6cd@localhost> sub -t /topic2/ -s
hallo2
hallo1

Publisher

mqttClient-MQTT_5_0-b8afd4bb-6e17-4297-bf6e-3f2c06c0960e@localhost> pub -t /topic2/ -m hallo2
mqttClient-MQTT_5_0-b8afd4bb-6e17-4297-bf6e-3f2c06c0960e@localhost> pub -t /topic1/ -m hallo1

Details

• Affected HiveMQ CLI version(s): 1.0
• Used JVM version: 11

CLI properties file

Add ability to set default configurations via properties file for the cli

• The file names hivemq-cli.default.properties

• The properties file is optional and overrides only defaults.

• The properties should be validated before loading

• For distributions/packaging - The properties file is located in the a folder .hivemq-cli in the user s home directory

• The properties can be set for overriding the following items

- mqtt version 
- debug level
- log file directory 
- default host
- default port
- default client-prefix 
- subscriber output 

Clients disconnect ungracefully

In the pub, sub and shell methods of the hivemq-cli the clients disconnect ungracefully, because the program terminates without disconnecting all the clients in the client cache. A shutdown-hook should be added to disconnect all clients from the cache, when the program terminates to prevent ungraceful disconnects.

Refactor HiveMQ-MQTT-Client library to be usable with GraalVM

The current mqtt-client lib is not usable within graalVM native-image,
because of reflection and dependenciy injection via guice and netty.

Second problem is the usage of netty - initializer must be declared within the library.

• talk to @SgtSilvio when MQTT lib is available for this

• make cli graalVM able

• add documentation for graalVM support

Refactor commands to not use default values

The Mqtt-Client API already handles default values if they are null.

Therefore it is not needed to give default values for the options in every command.

Example:
@CommandLine.Option(names = {"-wq", "--willQualityOfService"}, converter = MqttQosConverter.class, defaultValue = "AT_MOST_ONCE", description = "Quality of service level for the will message (default 0)") @Nullable private MqttQos willQos;

is already QoS level 1 by default of the Mqtt5WillPublish class.

Refactor / Fix Client cache

• Connect a disconnected client should be possible
• switch to a disconnected client should print an error

Using default Ssl configuration via -s in pub, sub and con produces NullPointerException

Expected behavior

mqtt> con -s
# Error message if the broker is not configured for default ssl
# Else connect the client with default ssl

Actual behavior

In Shell:

mqtt> con -s
# empty error log
mqttClient-MQTT_5_0-7b4ecf1d-da17-4e24-8ed6-c05840ed6f92@localhost>

-> Client is connected but without using ssl

In Logs:

java.lang.NullPointerException
	at com.hivemq.cli.commands.AbstractCommonFlags.doBuildSslConfig(AbstractCommonFlags.java:132)
	at com.hivemq.cli.commands.AbstractCommonFlags.buildSslConfig(AbstractCommonFlags.java:91)
	at com.hivemq.cli.commands.shell.ShellConnectCommand.run(ShellConnectCommand.java:77)
	at picocli.CommandLine.executeUserObject(CommandLine.java:1729)
	at picocli.CommandLine.access$900(CommandLine.java:145)
	at picocli.CommandLine$RunLast.handle(CommandLine.java:2101)
	at picocli.CommandLine$RunLast.handle(CommandLine.java:2068)
	at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:1935)
	at picocli.CommandLine.execute(CommandLine.java:1864)
	at com.hivemq.cli.commands.shell.ShellCommand.interact(ShellCommand.java:175)
	at com.hivemq.cli.commands.shell.ShellCommand.run(ShellCommand.java:123)
	at picocli.CommandLine.executeUserObject(CommandLine.java:1729)
	at picocli.CommandLine.access$900(CommandLine.java:145)
	at picocli.CommandLine$RunLast.handle(CommandLine.java:2101)
	at picocli.CommandLine$RunLast.handle(CommandLine.java:2068)
	at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:1935)
	at picocli.CommandLine.execute(CommandLine.java:1864)
	at com.hivemq.cli.MqttCLIMain.main(MqttCLIMain.java:73)

To Reproduce

Use pub, sub or con with the -s option

Details

• Affected MQTT CLI version(s): 1.0.0
• Used JVM version: 11.0.4 (AdoptOpenJDK OpenJDK 64-Bit Server VM 11.0.4+11)

Container Issue for small bugs

1. Documentation of password usage is not matching the usage
   -> musst be -P instead of -pw
2. debug mode is not working when using pipe symbol in command
   pub -t test -m “Hello” -up key1=value1|key2=value2 --debug
   -> do not use the pipe symbol for all the key/value arrays that are possible, as the pipe symbol will be also interpreted from the terminal itself
3. LOG behavior
   don't write information twice, as the log file should be interpreted as a WAL
   That means: depending on the Level, don't write lower level
4. write PUBACK / SUBACK instead of Receive in the Log file / Console
5. show default setting for host and port at start as information

Automatically update homebrew-mqtt-cli repository when merging into master

The travis.yml file should be updated so that the brew formula which is generated by the gradle task buildBrewFormula is automatically pushed to the homebrew repository.

Ability to get input parameter via STDIN

As described in README the password can be set via interactive input if a $ is used instead of a value.

Implement interactive input from STDIN

Example: pub -u test -pw $ -t xxx -m "my message"
$>>

Option naming for mqtt5 properties is confusing

The current naming of the options is a bit confusing.

In mosquitto a -D --properties flag is set to use properties:
See https://mosquitto.org/man/mosquitto_pub-1.html

Maybe use a similar approach.

Fix ArayIndex ooB in pub/sub if qos & topic amount dos not fit

AbstractMqttClientExecutor
publis / subscribe to several topix runs into AIooB Exception, if number of topics and qos dos not fit

External client disconnect is not recognized by cli

Expected behavior

If the client gets disconnected in the client context the cli should inform the user that the current context client is no longer available and change the context back to shell-context.

Actual behavior

The context client is shown, even though it got disconnected, and if a command with that client is used it will throw an error.

To Reproduce

In one terminal 1:
hivemq-cli shell
con -i client
client>

In terminal 2:
hivemq-cli shell
con -i client
E. g. hivemq will produce: Disconnecting already connected client with id client because another client connects with that id
client>

In terminal 1:
client> still shown, but it's actually not connected anymore.
client> pub -t topic -m msg will throw PUBLISH to topic failed with MQTT client is not connected.

Details

• Affected HiveMQ CLI version(s): 1.0
• Used JVM version: 11

Add ability to output received messages to file

Add another option to the Subscribe command which can be used to output the received messages to another file.

If using sub in non-shell mode print the output to STDOUT by default.

Add
@CommandLine.Option(names = {"-f", "--file"}, description = "The file to which the received messages will be written.")
private File receivedMessagesFile;

Examples:
hivemq-cli sub -t test -m "To File" -f output.txt
-> Writes received messages to file and stdout

hivemq-cli shell
> sub -t test -m "To File" -f output.txt
-> Writes received messages to file

Add Changes to README

Implement prefixIdentifier

@CommandLine.Option(names = {"-pi", "--prefixIdentifier"}, description = "The prefix of the client Identifier UTF-8 String")
private String prefixIdentifier;

A default prefixIdentifier should set to hmqClient and should also available in the default settings.

Add TLS and Client certificate support

Additional to the basic CONNECT - the user should be able to run the command with tls usage
and Client certificates

Add
--cafile : path to a file containing trusted CA certificates to enable encrypted certificate based communication.
--capath : path to a directory containing trusted CA certificates to enable encrypted communication.
--cert : client certificate for authentication, if required by server.
--key : client private key for authentication, if required by server.
--ciphers : openssl compatible list of TLS ciphers to support.
--tls-version : TLS protocol version, can be one of tlsv1.2 tlsv1.1 or tlsv1.3

Defaults to tlsv1.2 if available.

Add new options to README.adoc

Implement more Unit tests for MQTT packages

For some of the PUBLISH, SUBSCRIBE, CONNECT and DISCONNECT commands no unit tests exist.

Please add unit-tests for the missing Attributes in the MQTT packages.
Test success and wrong parameter settings and combinations.

Update and review the documentation

The documentation should be updated to reflect the current version of MQTT CLI.

• check if all options are named properly
• Unify the synopsis for all commands
• check if the synopsis of all commands are correct
• check if all links are working correctly
• Do not linebreak in Long Version options
• search for wrong naming

Make connect available in context mode

In context mode of the shell the user should be able to connect to a new client so unnecessary hopping between context and non-context is omitted.

Current Behavior:

con -i c1
c1> exit
con -i c2
c2>

Improved Behavior:

con -i c1
c1> con -i c2
c2>

Test Features of MQTT CLI

Try for each mode the available commands that are shown from the help menu
check and document:

• errors,
• missing and/or weird behavior,
• spelling errors
• check also the log files

Add default values to every usage help section

Add default values to every options description.

Implement VERBOSE Mode

It should be possible to start MQTT-CLI in verbose mode, so that each command will be logged including MQTT Message information.

Add parameter for Abstract Command ( -vb --verbose )
Logging of all CLI Commands
Logging of MQTT Messages on Protocol level
Logging of caching

Add Unsubscribe

Add ability to unsubscribe from a given topic(s)

• add Subcommand (only in shell mode) after connecting a client
• if the client is not subscribed, this will be ignored

Client Cache does not remove items

ClientCache<String, String> cache = new ClientCache();
cache.put("a", "b")
cache.remove("a")
cache.hasKey("a")-> true. Key was only removed in the internally used linkedList but not in the actual HashMap which is used to retrieve the data