fhmq / hmq

High performance mqtt broker

Free and High Performance MQTT Broker

About

Golang MQTT Broker, Version 3.1.1, and Compatible for eclipse paho client and mosquitto-client

RUNNING

$ go get github.com/fhmq/hmq
$ cd $GOPATH/github.com/fhmq/hmq
$ go run main.go

Usage of hmq:

Usage: hmq [options]

Broker Options:
    -w,  --worker <number>            Worker num to process message, perfer (client num)/10. (default 1024)
    -p,  --port <port>                Use port for clients (default: 1883)
         --host <host>                Network host to listen on. (default "0.0.0.0")
    -ws, --wsport <port>              Use port for websocket monitoring
    -wsp,--wspath <path>              Use path for websocket monitoring
    -c,  --config <file>              Configuration file

Logging Options:
    -d, --debug <bool>                Enable debugging output (default false)
    -D                                Debug enabled

Cluster Options:
    -r,  --router  <rurl>             Router who maintenance cluster info
    -cp, --clusterport <cluster-port> Cluster listen port for others

Common Options:
    -h, --help                        Show this message

hmq.config

{
	"workerNum": 4096,
	"port": "1883",
	"host": "0.0.0.0",
	"cluster": {
		"host": "0.0.0.0",
		"port": "1993"
	},
	"router": "127.0.0.1:9888",
	"wsPort": "1888",
	"wsPath": "/ws",
	"wsTLS": true,
	"tlsPort": "8883",
	"tlsHost": "0.0.0.0",
	"tlsInfo": {
		"verify": true,
		"caFile": "tls/ca/cacert.pem",
		"certFile": "tls/server/cert.pem",
		"keyFile": "tls/server/key.pem"
	},
	"plugins": {
		"auth": "authhttp",
		"bridge": "kafka"
	}
}

Features and Future

• Supports QOS 0 and 1

• Cluster Support

• Containerization

• Supports retained messages

• Supports will messages

• Websocket Support

• TLS/SSL Support

• Auth Support

  • Auth Connect
  • Auth ACL
  • Cache Support

• Kafka Bridge Support

  • Action Deliver
  • Regexp Deliver

• HTTP API

  • Disconnect Connect (future more)

Share SUBSCRIBE

| Prefix              | Examples                                  | Publish                      |
| ------------------- |-------------------------------------------|--------------------------- --|
| $share/<group>/topic  | mosquitto_sub -t ‘$share/<group>/topic’ | mosquitto_pub -t ‘topic’     |

Cluster

 1, start router for hmq  (https://github.com/fhmq/router.git)
 	$ go get github.com/fhmq/router
 	$ cd $GOPATH/github.com/fhmq/router
 	$ go run main.go
 2, config router in hmq.config  ("router": "127.0.0.1:9888")
 

Other Version Of Cluster Based On gRPC: click here

Online/Offline Notification

 topic:
     $SYS/broker/connection/clients/<clientID>
 payload:
	{"clientID":"client001","online":true/false,"timestamp":"2018-10-25T09:32:32Z"}

Performance

• High throughput

• High concurrency

• Low memory and CPU

License

• Apache License Version 2.0

Reference

• Surgermq.(https://github.com/zentures/surgemq)

Benchmark Tool

• https://github.com/inovex/mqtt-stresser
• https://github.com/krylovsk/mqtt-benchmark

hmq's Issues

Losing messages

I have tried a simple test as follows on a Mac with the go1.11.5:

Start the broker:
go run main.go -p 8888

Start a subscriber:
mosquitto_sub -h localhost -p 8888 -t 'test/room' -v

Publish messages:
mosquitto_pub -h localhost -p 8888 -t 'test/room' -m "$(date)"

Results are that only every third of fourth message are displayed on the subscribe window.

The mosquitto package is on a Mac:
mosquitto_pub version 1.5.4 running on libmosquitto 1.5.4.

Elastic Message pool and Worker pool?

I was thinking of introducing some logic to dynamically size the message pool and the worker pool.
Something like: the pool start at a low number and when it's not far from being full, increase the size of the pool. The logic of decreasing to pool would be the same.
What do you think about it?
Advantages I can think of:

• quicker startup
• lower the memory footprint
• adapt automatically to the computer capacity it's running on
• less things to worry about in configuration file

panic: runtime error: invalid memory address or nil pointer dereference

{"level":"error","ts":1523360297.1253836,"logger":"hmq.Broker","caller":"broker/client.go:251","msg":"process message for psub error, ","error":"write tcp 172.31.58.23:10086->14.26.19.135:25134: use of closed network connection","ClientID":"A1020180316031600"}
{"level":"error","ts":1523360297.1254902,"logger":"hmq.Broker","caller":"broker/client.go:251","msg":"process message for psub error, ","error":"write tcp 172.31.58.23:10086->14.26.19.135:25134: use of closed network connection","ClientID":"A1020180316031600"}
{"level":"error","ts":1523360297.125594,"logger":"hmq.Broker","caller":"broker/client.go:251","msg":"process message for psub error, ","error":"write tcp 172.31.58.23:10086->14.26.19.135:25134: use of closed network connection","ClientID":"A1020180316031600"}
{"level":"error","ts":1523360297.1257198,"logger":"hmq.Broker","caller":"broker/client.go:251","msg":"process message for psub error, ","error":"write tcp 172.31.58.23:10086->14.26.19.135:25134: use of closed network connection","ClientID":"A1020180316031600"}
{"level":"error","ts":1523360297.125832,"logger":"hmq.Broker","caller":"broker/client.go:251","msg":"process message for psub error, ","error":"write tcp 172.31.58.23:10086->14.26.19.135:25134: use of closed network connection","ClientID":"A1020180316031600"}
{"level":"error","ts":1523360297.146843,"logger":"hmq.Broker","caller":"broker/client.go:127","msg":"read packet error: ","error":"read tcp 172.31.58.23:10086->14.26.19.135:25134: use of closed network connection","ClientID":"A10201803160316"}
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x661b4a]

goroutine 1468 [running]:
panic(0x6d07e0, 0x8db7b0)
C:/Go/src/runtime/panic.go:554 +0x3c1 fp=0xc421337278 sp=0xc4213371d8 pc=0x429551
runtime.panicmem()
C:/Go/src/runtime/panic.go:63 +0x5e fp=0xc421337298 sp=0xc421337278 pc=0x42842e
runtime.sigpanic()
C:/Go/src/runtime/signal_unix.go:388 +0x17a fp=0xc4213372e8 sp=0xc421337298 pc=0x43e6ca
github.com/eclipse/paho.mqtt.golang/packets.(*PublishPacket).Write(0xc421220440, 0x0, 0x0, 0x0, 0x0)
C:/Users/lukz/go/src/github.com/eclipse/paho.mqtt.golang/packets/publish.go:39 +0x25a fp=0xc421337430 sp=0xc4213372e8 pc=0x661b4a
github.com/fhmq/hmq/broker.(*client).WriterPacket(0xc420c1b590, 0x771f40, 0xc421220440, 0xc420f93c50, 0x905f60)
E:/go/src/github.com/fhmq/hmq/broker/client.go:534 +0xb2 fp=0xc421337488 sp=0xc421337430 pc=0x678a72
github.com/fhmq/hmq/broker.(*client).ProcessPublishMessage(0xc420c1a5a0, 0xc421220440)
E:/go/src/github.com/fhmq/hmq/broker/client.go:249 +0x124 fp=0xc421337868 sp=0xc421337488 pc=0x675a24
github.com/fhmq/hmq/broker.(*client).ProcessPublish(0xc420c1a5a0, 0xc421220440)
E:/go/src/github.com/fhmq/hmq/broker/client.go:198 +0x87c fp=0xc421337d70 sp=0xc421337868 pc=0x6755ec
github.com/fhmq/hmq/broker.ProcessMessage(0xc420f117e0)
E:/go/src/github.com/fhmq/hmq/broker/client.go:157 +0x574 fp=0xc421337f70 sp=0xc421337d70 pc=0x674c14
github.com/fhmq/hmq/broker.(*Broker).StartDispatcher.func1.1()
E:/go/src/github.com/fhmq/hmq/broker/broker.go:102 +0x2a fp=0xc421337f88 sp=0xc421337f70 pc=0x68097a
github.com/fhmq/hmq/pool.startWorker.func1(0xc420018300, 0xc4200523c0)
E:/go/src/github.com/fhmq/hmq/pool/pool.go:160 +0x68 fp=0xc421337fd0 sp=0xc421337f88 pc=0x665ed8
runtime.goexit()
C:/Go/src/runtime/asm_amd64.s:2361 +0x1 fp=0xc421337fd8 sp=0xc421337fd0 pc=0x456811
created by github.com/fhmq/hmq/pool.startWorker
E:/go/src/github.com/fhmq/hmq/pool/pool.go:145 +0x49

Can some one point good documentation for this awesome project?

Thanks for the awesome broker in golang.
I do not see good documentation for this project.
can someone provide the link for a document?

can not 100% completed with mqtt-stresser

https://github.com/inovex/mqtt-stresser

error default path for kafka plugins

Environment

• hmq version: 68cd5e9
• go version: go1.13.3
• OS: MacOsx
• Installed from: source

Bug description

hmq/plugins/bridge/kafka.go

Line 32 in 68cd5e9

content, err := ioutil.ReadFile("./plugins/mq/kafka/kafka.json")

the config for plugin kafka is at $PROJECT_ROOT/plugins/bridge/kafka/kafka.json , but the default still be $PROJECT_ROOT/plugins/mq/kafka/kafka.json

For cluster

I am wondering if the hmq supports distributed deploying? what does router do? If I understand right, the router part only collects the brokers' url list in the cluster. What about the load balance?

Remove a go routine

I think we should remove the following go routine as we are already are in a go routine and subsequent process are against the same message queue.
What do you think about that one?

hmq/broker/dispatcher.go

Line 37 in 221d004

go func(msg *Message) {

multilevel wildcard does not match empty subtree

The multi-level wildcard # is not matching an empty subtree. In other words:
A subscription to
/zoo/animal/#

should match

/zoo/animal/zebra
/zoo/animal/elephant
/zoo/animal/bengal/tiger

and it should also match
/zoo/animal
fhmq is not currently handling this case correctly, ie, is not matching /zoo/animal

The following test, which you can run in broker/lib/topics illustrates. (The file is a .go, changed to .txt so could attach)
memtopics_test.txt

panic on ProcessPublishMessage

Hi,
I'm not sure what can cause this issue but sometimes when a client appear to somehow disconnect I'm getting a bunch of:

ERROR   broker  broker/client.go:252    process message for psub error,         {"error": "write tcp 127.0.0.1:1888->127.0.0.1:41254: use of closed network connection", "ClientID": "e0d159c9-0bc3-4f88-8e96-e5a33fa4d1dd"} 
github.com/fhmq/hmq/broker.(*client).ProcessPublishMessage
        github.com/fhmq/hmq/broker/client.go:252

which seems to terminate at some point with a panic:

[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x91906a]

goroutine 5798521 [running]:
bitbucket.org/digitalisio/axonops/vendor/github.com/eclipse/paho.mqtt.golang/packets.(*PublishPacket).Write(0xc5142c6f40, 0x0, 0x0, 0x0, 0x0)
        github.com/eclipse/paho.mqtt.golang/packets/publish.go:39 +0x25a
bitbucket.org/digitalisio/axonops/vendor/github.com/fhmq/hmq/broker.(*client).WriterPacket(0xc420599a40, 0xd74a00, 0xc5142c6f40, 0x0, 0x0)
        github.com/fhmq/hmq/broker/client.go:428 +0x11c
bitbucket.org/digitalisio/axonops/vendor/github.com/fhmq/hmq/broker.(*client).ProcessPublishMessage(0xc423370c80, 0xc5142c6f40)
        github.com/fhmq/hmq/broker/client.go:250 +0x1c9
bitbucket.org/digitalisio/axonops/vendor/github.com/fhmq/hmq/broker.(*client).ProcessPublish(0xc423370c80, 0xc5142c6f40)
        github.com/fhmq/hmq/broker/client.go:185 +0x587
bitbucket.org/digitalisio/axonops/vendor/github.com/fhmq/hmq/broker.ProcessMessage(0xc486b327e0)
        github.com/fhmq/hmq/broker/client.go:144 +0x2dd
bitbucket.org/digitalisio/axonops/vendor/github.com/fhmq/hmq/broker.(*Broker).SubmitWork.func1()
        github.com/fhmq/hmq/broker/broker.go:115 +0x2a
bitbucket.org/digitalisio/axonops/vendor/github.com/fhmq/hmq/pool.startWorker.func1(0xc4203a64e0, 0xc4202e49c0)
        github.com/fhmq/hmq/pool/pool.go:160 +0x68
created by bitbucket.org/digitalisio/axonops/vendor/github.com/fhmq/hmq/pool.startWorker
        github.com/fhmq/hmq/pool/pool.go:145 +0x49

Regards

Question about shutting down broker gracefully

@chowyu08 @MarcMagnin
As the title, how to handle the broker shutdown/restart? The alive clients need to be notified one by one?

thanks

hmq 是否支持集群搭建

hmq Whether to support cluster building

License ?

What is the license for this project? Can you add a license file to it? Thank you!

Regression: Not Authorized when auth not specified and username blank

The recent work on adding auth.go seems to have caused a regression, trying to connect to the broker returns an error "Not Authorized"

In broker.go handleConnection()

   if typ == CLIENT && !b.CheckConnectAuth(string(msg.ClientIdentifier), string(msg.Username), string(msg.Password)) {
                connack.ReturnCode = packets.ErrRefusedNotAuthorised

Unfortunately CheckConnAuth() is not checking for the auth to be the mock auth:

func (b *Broker) CheckConnectAuth(clientID, username, password string) bool {
        if b.auth != nil {
                if clientID == "" || username == "" {
                        return false
                }
                return b.auth.CheckConnect(clientID, username, password)
        }

        return true

}

In this case b.auth is not nil because it's the mock auth, and clientID and username are blank so "return false". The mock auth CheckConnect() always returns true which is what we want in this case.

Workaround - make sure client sets the client id and username to something.

distributed lock need

when client connecting, need a distributed lock

用一段时间后会自动挂掉。呕血，连续几次了。邮件又没人回。哎

用一段时间后会自动挂掉。呕血，连续几次了。邮件又没人回。哎

Can't do "go get ..."

Hi,

Thanks a lot for this project!
I'd like to use your project as a dependency with "go get github.com/fhmq/hmq" but I got the following error:

package hmq/broker: unrecognized import path "hmq/broker" (import path does not begin with hostname)

Any idea on that one?
Many thanks

when Support QOS 2？

when Support QOS 2 ?

WS: use of closed network connection

Hi,

I'd like to use the WS connection but I got the following errors when connecting up (broker output):

Time:2018-01-16 17:48:12        file:client.go  level:ERROR     read packet error: read tcp 127.0.0.1:1888->127.0.0.1:35916: use of closed network connection clientID = f27c46e3-8d81-4b62-b405-e9b5b2149f9f
Time:2018-01-16 17:48:12        file:client.go  level:DEBUG     Recv message: DISCONNECT: dup: false qos: 0 retain: false rLength: 0 clientID = f27c46e3-8d81-4b62-b405-e9b5b2149f9f
Time:2018-01-16 17:48:12        file:broker.go  level:WARN      client exist, close old... clientID = f27c46e3-8d81-4b62-b405-e9b5b2149f9f
Time:2018-01-16 17:48:12        file:client.go  level:DEBUG     Recv message: DISCONNECT: dup: false qos: 0 retain: false rLength: 0 clientID = f27c46e3-8d81-4b62-b405-e9b5b2149f9f
Time:2018-01-16 17:48:12        file:client.go  level:ERROR     read packet error: read tcp 127.0.0.1:1888->127.0.0.1:35918: use of closed network connection clientID = f27c46e3-8d81-4b62-b405-e9b5b2149f9f
Time:2018-01-16 17:48:12        file:client.go  level:DEBUG     Recv message: DISCONNECT: dup: false qos: 0 retain: false rLength: 0 clientID = f27c46e3-8d81-4b62-b405-e9b5b2149f9f

I'm connecting using the pahoMQTT client like the following:

opts := pahoMQTT.NewClientOptions()
opts.AddBroker("ws://0.0.0.0:1888/ws")
opts.SetClientID("client-test")
opts.SetKeepAlive(time.Duration(120 * time.Second))
opts.SetConnectTimeout(time.Duration(5 * time.Second))

mqttClient := pahoMQTT.NewClient(opts)
token := mqttClient.Connect()
if token.WaitTimeout(time.Duration(5000*time.Millisecond)) || token.Error() != nil {
	log.Panic("Unable to connect to MQTT. ", token.Error())
}

Btw token.Error() == nil

Many thanks.

[Question] Is the router could be part of the broker?

Hey,

So if I understand it, a cluster of broker require a router. That sounds like a single point of failure. Is this router functionality could be embedded in the broker such as I suppose emqtt is doing? http://emqtt.io/docs/v2/cluster.html

Regards

allow http monitor port to be configurable, currently hardwired to 8080

InitHTTPMoniter() currently does:

router.Run(":8080")

The port value needs to be configurable

Last commit broke the build

Hello, the last commit: 896769f broke the build. As Auth interface has CheckACL(action, clientID, username, ip, topic string) bool method, however broker/auth.go:28 uses b.auth.CheckACL(action, username, topic). When you build it with go build . you get:
# github.com/fhmq/hmq/broker broker/auth.go:28:25: not enough arguments in call to b.auth.CheckACL have (string, string, string) want (string, string, string, string, string)

Does it support "clean session = false"?

Does it support "clean session = false"?

Not working after a period of time

Not working after a period of time

Question about message publishing with QoS1

hmq/broker/client.go

Line 200 in 4750091

case QosAtLeastOnce:

@chowyu08 @MarcMagnin
I am not sure which step should be at first , puback or publish to subscriber? thanks

Question about the struct 'subscription'

@chowyu08 @MarcMagnin
A newer for mqtt. I am curious what means the member 'queue' in the struct 'subscription' is? thanks
true --> ?
false --> ?

type subscription struct {
client *client
topic string
qos byte
queue bool
}

Does it support "will message"?

go get fails; go run fails

go get github.com/fhmq/hmq
package context: unrecognized import path "context" (import path does not begin with hostname)

But it downloads fine. Then:

go run main.go 
# github.com/fhmq/hmq/plugins/authhttp
plugins/authhttp/authhttp.go:51:19: unknown field 'MaxConnsPerHost' in struct literal of type http.Transport

This is running on Ubuntu 18.04, go version go1.10.4 linux/amd64.

Bit of a go newb, so not sure whether these are configuration errors or something else.

error here when build

[root@iz2zeamz4btzv5e42dk6hsz mqtt]# go run main.go

go.uber.org/zap

/usr/local/go/src/go.uber.org/zap/field.go:10: syntax error: unexpected = in type declaration

不错，可以搭建集群么?

可以部署集群么？

hi,have some bug,please fix

github.com/fhmq/hmq/broker.(*client).readLoop
/home/yu/gowork/src/github.com/fhmq/hmq/broker/client.go:130
github.com/fhmq/hmq/broker.(*Broker).handleConnection
/home/yu/gowork/src/github.com/fhmq/hmq/broker/broker.go:339
2018-07-24T17:01:19.010+0800 DEBUG hmq.Broker broker/client.go:153 Recv message from client, {"ClientID": "iOS"}
2018-07-24T17:01:38.107+0800 DEBUG hmq.Broker broker/client.go:153 Recv message from client, {"ClientID": "iOS"}
2018-07-24T17:01:38.107+0800 ERROR hmq.Broker broker/sublist.go:214 server/sublist.go: {"error": "Topic format error with index of //"}
github.com/fhmq/hmq/broker.(*Sublist).Match
/home/yu/gowork/src/github.com/fhmq/hmq/broker/sublist.go:214
github.com/fhmq/hmq/broker.(*client).ProcessPublishMessage
/home/yu/gowork/src/github.com/fhmq/hmq/broker/client.go:239
github.com/fhmq/hmq/broker.(*client).ProcessPublish
/home/yu/gowork/src/github.com/fhmq/hmq/broker/client.go:209
github.com/fhmq/hmq/broker.ProcessMessage
/home/yu/gowork/src/github.com/fhmq/hmq/broker/client.go:160
github.com/fhmq/hmq/broker.Worker.Start.func1
/home/yu/gowork/src/github.com/fhmq/hmq/broker/worker.go:26
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0x690101]

goroutine 1013 [running]:
github.com/fhmq/hmq/broker.(*client).ProcessPublishMessage(0xc4200ee1e0, 0xc420b93240)
/home/yu/gowork/src/github.com/fhmq/hmq/broker/client.go:241 +0xa1
github.com/fhmq/hmq/broker.(*client).ProcessPublish(0xc4200ee1e0, 0xc420b93240)
/home/yu/gowork/src/github.com/fhmq/hmq/broker/client.go:209 +0x549
github.com/fhmq/hmq/broker.ProcessMessage(0xc420b98460)
/home/yu/gowork/src/github.com/fhmq/hmq/broker/client.go:160 +0x445
github.com/fhmq/hmq/broker.Worker.Start.func1(0xc4208f4600, 0xc420c3aba0, 0xc420c3ac00)
/home/yu/gowork/src/github.com/fhmq/hmq/broker/worker.go:26 +0x3f
created by github.com/fhmq/hmq/broker.Worker.Start
/home/yu/gowork/src/github.com/fhmq/hmq/broker/worker.go:19 +0x53

Enable use of MQTT broker as a library

It would be nice to be able to use the broker as a dep as well.
I'll look into how to make it generic enough to satisfy such requirement.
@chowyu08 what do you think of it?

[bug?]routes->remotes

In broker/broker.go, does it should be b.remotes instead of b.routes? since the client objects in b.routes do not have an initialized session object, which will lead to a panic.

func (b *Broker) BroadcastSubOrUnsubMessage(packet packets.ControlPacket) {
	b.remotes.Range(func(key, value interface{}) bool {
		r, ok := value.(*client)
		if ok {
			r.WriterPacket(packet)
		}
		return true
	})
	// log.Info("BroadcastSubscribeMessage remotes: ", s.remotes)
}

Panic with large numbers of subscribers and producers !

I wrote a client program based on paho in golang. It starts multiple publishers & subcribers using goroutines.
When the broker is running on the server, this panic appears once I increase pub/sub numbers ( once we reach a few thousands the error appears fairly frequently )
At first, the program works and some messages are correctly received by the subscribers. But then, the broker throws this panic in the middle of the program.

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x665daa]

goroutine 10903 [running]:
github.com/eclipse/paho.mqtt.golang/packets.(*PublishPacket).Write(0xc4229880c0, 0x0, 0x0, 0x0, 0x0)
/home/ihsen/Documents/Hmq/src/github.com/eclipse/paho.mqtt.golang/packets/publish.go:39 +0x25a
github.com/fhmq/hmq/broker.(*client).WriterPacket(0xc4213045a0, 0x7781c0, 0xc4229880c0, 0xc423bebd00, 0x2)
/home/ihsen/Documents/Hmq/src/github.com/fhmq/hmq/broker/client.go:535 +0xb2
github.com/fhmq/hmq/broker.(*client).ProcessPublishMessage(0xc425ba90e0, 0xc4229880c0)
/home/ihsen/Documents/Hmq/src/github.com/fhmq/hmq/broker/client.go:250 +0x124
github.com/fhmq/hmq/broker.(*client).ProcessPublish(0xc425ba90e0, 0xc4229880c0)
/home/ihsen/Documents/Hmq/src/github.com/fhmq/hmq/broker/client.go:199 +0x8a1
github.com/fhmq/hmq/broker.ProcessMessage(0xc421af20c0)
/home/ihsen/Documents/Hmq/src/github.com/fhmq/hmq/broker/client.go:158 +0x574
github.com/fhmq/hmq/broker.(*Broker).StartDispatcher.func1.1()
/home/ihsen/Documents/Hmq/src/github.com/fhmq/hmq/broker/broker.go:102 +0x2a
github.com/fhmq/hmq/pool.startWorker.func1(0xc4201cc000, 0xc42009e3c0)
/home/ihsen/Documents/Hmq/src/github.com/fhmq/hmq/pool/pool.go:160 +0x68
created by github.com/fhmq/hmq/pool.startWorker
/home/ihsen/Documents/Hmq/src/github.com/fhmq/hmq/pool/pool.go:145 +0x49

Not long before the crash, we observe some broken pipe errors, but we do not know if the errors are connected with the crash.
{"level":"error","ts":1523275251.1890993,"logger":"hmq.Broker","caller":"broker/client.go:252","msg":"process message for psub error, ","error":"write tcp 127.0.0.1:1883->127.0.0.1:40269: write: broken pipe","ClientID":"client107"}

Any idea of what is wrong with this ?

PS : I used hmq master branch , go version 1.10 and paho v1.1.1.

[Question] What about using Zap logger?

Hey,
Zap logger is a pretty fast / light logger and I think it would be a good fit for an MQTT broker:
https://github.com/uber-go/zap

Let me know what you think about it. I could raise a PR to implement it.

Regards

Why is this line commented out?

hmq/broker/lib/topics/memtopics.go

Line 487 in 68cd5e9

// if qos >= this.qos[i] {

how could we support QOS 2?

In our project we need to support QOS 2, so anyone else who know how to support QOS 2？Who can help me?

does it support device online and offline

when device is online or offline , there is a topic send a message ?

Can't we deal with a chan instead of an array of MessagePool?

hmq/broker/msgpool.go

Line 24 in 221d004

MSGPool []MessagePool

I was wondering if we couldn't replace the MessagePool array by a chan?
This would enable us to remove a bunch of goroutines in the dispatcher.

Question about qos1 implement

at client.go , function processClientPublish

when it`s QosAtLeastOnce, we only send puback to sender and send the pack to receiver.

What will we do if this send failed?
As my understanding, we should save this package and delete it when we receive puback.
If timeout for puback, we should resend it.

But I found nothing about it.

pub=1 sub=2?

acl config file ：pub=1 sub=2
but now authhttp acl pub=2,sub=1

Can you give a conf of auth user password

Can you give a conf of auth user password

subscribe multiple times with the same client

It is possible to subscribe multiple times with the same client on the same topic.
To reproduce, subscribe once. Then hard stop the client and restart the client. The problem is that after such scenario, dup messages will be sent to that client.
I think we need a mechanism to prevent a client to subscribe multiple times to the same topic.

Prevent message output as some can be quite large with binary in?

hmq/broker/client.go

Line 152 in d9e6e21

log.Debug("Recv message: ", ca.String(), " clientID = ", c.info.clientID)

I'm using some large binary messages so I'd like to remove that String() call that show the content of the message. What do you think?

Make the config as a parameter of the broker instance

Hey,

It would be nice to be able to pass the config path to the broker LoadConfig.

Regards

Persist retains in boltDB?

I was thinking of persisting the retained messages in boltDB.
Any idea about persisting feature?

Data race detected when killing client application

Hey,

While running the app with the -race flag, when I kill a subscribed client, I receive this from the broker:

...
1516804674835374285 [Error] Error trying to connect to route: dial tcp 127.0.0.1:9888: getsockopt: connection refused
1516804674836070893 [Debug] Connect to route timeout ,retry...
1516804678944420663 [Debug] Recv message: clientID = test
1516804683941713807 [Debug] Recv message: clientID = test
1516804688945398091 [Debug] Recv message: clientID = test
1516804693942847670 [Debug] Recv message: clientID = test
1516804694838098979 [Error] Error trying to connect to route: dial tcp 127.0.0.1:9888: getsockopt: connection refused
1516804694838582526 [Debug] Connect to route timeout ,retry...
1516804698944257105 [Debug] Recv message: clientID = test
1516804699547275423 [Error] read packet error: EOF clientID = test
1516804699550136489 [Debug] Recv message: clientID = test
==================
WARNING: DATA RACE
Write at 0x00c4201401a0 by goroutine 809:
  github.com/MarcMagnin/hmq/broker.(*client).Close()
     github.com/MarcMagnin/hmq/broker/client.go:491 +0xbb
  github.com/MarcMagnin/hmq/broker.ProcessMessage()
     github.com/MarcMagnin/hmq/broker/client.go:176 +0x430
  github.com/MarcMagnin/hmq/broker.Worker.Start.func1()
     github.com/MarcMagnin/hmq/broker/worker.go:24 +0x4f

Previous read at 0x00c4201401a0 by goroutine 808:
  github.com/MarcMagnin/hmq/broker.(*client).ProcessPublish()
     github.com/MarcMagnin/hmq/broker/client.go:183 +0x57
  github.com/MarcMagnin/hmq/broker.ProcessMessage()
     github.com/MarcMagnin/hmq/broker/client.go:159 +0x461
  github.com/MarcMagnin/hmq/broker.Worker.Start.func1()
     github.com/MarcMagnin/hmq/broker/worker.go:24 +0x4f

Goroutine 809 (running) created at:
  github.com/MarcMagnin/hmq/broker.Worker.Start()
     github.com/MarcMagnin/hmq/broker/worker.go:17 +0x60
  github.com/MarcMagnin/hmq/broker.(*Dispatcher).Run()
     github.com/MarcMagnin/hmq/broker/dispatcher.go:19 +0xa4
  github.com/MarcMagnin/hmq/broker.StartDispatcher()
     github.com/MarcMagnin/hmq/broker/dispatcher.go:12 +0xb1
  github.com/MarcMagnin/hmq/broker.(*Broker).Start()
     github.com/MarcMagnin/hmq/broker/broker.go:72 +0x41

Goroutine 808 (running) created at:
  github.com/MarcMagnin/hmq/broker.Worker.Start()
     github.com/MarcMagnin/hmq/broker/worker.go:17 +0x60
  github.com/MarcMagnin/hmq/broker.(*Dispatcher).Run()
     github.com/MarcMagnin/hmq/broker/dispatcher.go:19 +0xa4
  github.com/MarcMagnin/hmq/broker.StartDispatcher()
     github.com/MarcMagnin/hmq/broker/dispatcher.go:12 +0xb1
  github.com/MarcMagnin/hmq/broker.(*Broker).Start()
     github.com/MarcMagnin/hmq/broker/broker.go:72 +0x41
==================

not use docker, websocket can't use

I use ubuntu , go build main.go and then run ./main &
but websocket can't use

panic “Topic format error with index of //”

{"level":"error","ts":1523278658.266067,"logger":"hmq.Broker","caller":"broker/sublist.go:213","msg":"\tserver/sublist.go: ","error":"Topic format error with index of //"}
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x673539]

[Question] Diagram of the application

Hey,

Would it be possible to design a diagram of the application as I'm trying to get my head around?
Many thanks.