FYI: https://github.com/uber-go/zap

ssl ready for plain tcp and ws? if ready, can u add examples for them?

Errors are important for development. There is one of best pratices for golang errors. Reference: Don’t just check errors, handle them gracefully.

This need a package github.com/pkg/errors, and I think it also very useful for this project.

func (s *Type) Start() {
	s.onStart.Do(func() {
		s.txRun()
		s.EventPoll.Start(s.Desc, s.rxRun) // nolint: errcheck
		s.started.Done()
	})
}

panic: file tcp 127.0.0.1:1883->127.0.0.1:1691: not supported by windows
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xc0000005 code=0x0 addr=0x0 pc=0x710bca]

goroutine 33 [running]:
github.com/VolantMQ/volantmq/connection.(*Type).Start(0x0)
E:/LzGo/volantmq/src/github.com/VolantMQ/volantmq/connection/connection.go:217 +0x3a
github.com/VolantMQ/volantmq/clients.(*session).start(0xc04216e000)
E:/LzGo/volantmq/src/github.com/VolantMQ/volantmq/clients/session.go:142 +0x93
github.com/VolantMQ/volantmq/clients.(*Manager).NewSession.func1(0xc042177b40, 0xc04223dc48, 0xc0420ea000, 0xc042177b30, 0xc042177b28, 0xc042177b20)
E:/LzGo/volantmq/src/github.com/VolantMQ/volantmq/clients/sessions.go:160 +0x4b7
panic(0x891840, 0xc04265a0f0)
E:/Go/src/runtime/panic.go:491 +0x291
github.com/troian/easygo/netpoll.Must(0x0, 0xb5d840, 0xc04265a0f0, 0xb5d840)
E:/LzGo/volantmq/src/github.com/troian/easygo/netpoll/handle.go:41 +0x5b
github.com/VolantMQ/volantmq/clients.(*Manager).newConnectionPreConfig(0xc0420ea000, 0xc042177c48, 0x0)
E:/LzGo/volantmq/src/github.com/VolantMQ/volantmq/clients/sessions.go:299 +0x99
github.com/VolantMQ/volantmq/clients.(*Manager).configureSession(0xc0420ea000, 0xc042177c48, 0xc04216e000, 0xc042088060, 0x20, 0xc04216e000, 0x0, 0x0, 0x5594a0294570c68c)
E:/LzGo/volantmq/src/github.com/VolantMQ/volantmq/clients/sessions.go:322 +0x14c
github.com/VolantMQ/volantmq/clients.(*Manager).NewSession(0xc0420ea000, 0xc04223dc48)
E:/LzGo/volantmq/src/github.com/VolantMQ/volantmq/clients/sessions.go:181 +0x22c
github.com/VolantMQ/volantmq/transport.(*baseConfig).handleConnection(0xc042042000, 0xb65f20, 0xc04208a040)
E:/LzGo/volantmq/src/github.com/VolantMQ/volantmq/transport/base.go:147 +0xb2d
github.com/VolantMQ/volantmq/transport.(*tcp).Serve.func1(0xc042042000, 0xb660a0, 0xc042162000)
E:/LzGo/volantmq/src/github.com/VolantMQ/volantmq/transport/tcp.go:136 +0xfb
created by github.com/VolantMQ/volantmq/transport.(*tcp).Serve
E:/LzGo/volantmq/src/github.com/VolantMQ/volantmq/transport/tcp.go:130 +0xda

When any error occurs in clients/session.go, the library panics because the logger hasn't been instantiated.

goroutine 21 [running]:
go.uber.org/zap.(*Logger).check(0x0, 0x2, 0x463d310, 0x1f, 0xc42105f4c0)
/Users/mochi/Development/Go/src/go.uber.org/zap/logger.go:266 +0xfd
go.uber.org/zap.(*Logger).Error(0x0, 0x463d310, 0x1f, 0xc42105f4c0, 0x1, 0x1)
/Users/mochi/Development/Go/src/go.uber.org/zap/logger.go:202 +0x44
github.com/VolantMQ/volantmq/clients.(*session).SignalUnSubscribe(0xc4200a4000, 0xc4212b87e0, 0xc420febcd7, 0xc420018432, 0x1, 0xc420febd50)
/Users/mochi/Development/Go/src/github.com/VolantMQ/volantmq/clients/session.go:234 +0x34e
github.com/VolantMQ/volantmq/connection.(*impl).processIncoming(0xc420098500, 0x4945760, 0xc4212b87e0, 0xc4212b87e0, 0x0)
/Users/mochi/Development/Go/src/github.com/VolantMQ/volantmq/connection/connection.go:633 +0x437
github.com/VolantMQ/volantmq/connection.(*impl).rxRoutine(0xc420098500)
/Users/mochi/Development/Go/src/github.com/VolantMQ/volantmq/connection/receiver.go:59 +0x24b
github.com/VolantMQ/volantmq/connection.(*impl).rxRun.func1(0xc420098500)
/Users/mochi/Development/Go/src/github.com/VolantMQ/volantmq/connection/receiver.go:22 +0x5e
created by github.com/VolantMQ/volantmq/connection.(*impl).rxRun
/Users/mochi/Development/Go/src/github.com/VolantMQ/volantmq/connection/receiver.go:19 +0x9d
exit status 2

It appears that when a newSession is created from the session manager, no instance of Zap logger is transferred from the manager.

The problem can be resolved by adding ses.log = m.log to sessions.go:434, however since I don't know enough about the current and planned architecture for VolantMQ I haven't issued a request. Maybe @troian can advise?

(fantastic work on all this, by the way).

func (m *Manager) allocContainer(id string, createdAt time.Time, cn connection.Session) *container {
ses := newSession(sessionPreConfig{
id: id,
createdAt: createdAt,
conn: cn,
messenger: m.TopicsMgr,
persistence: m.persistence,
})

wrap := &container{}
ses.idLock = &wrap.lock
ses.log = m.log // <-- :434 transfer manager logger to session
wrap.ses.Store(ses)
wrap.acquire()

return wrap
}

TBD

Seems it was introduced in last push, but there is no PersistedPackets type in persistence package

don't want persistence anything
or if mqtt client library bug,then don't reply such as PUBCOMP,broker memory will leak?

so want a empty Persistence provider

• Review if current state is appropriate to be clustered
• Write requirements

Is it possible to add some middleware option like in the HTTP package? For our project I need to process incoming messages before sending it to the subscribers.

Code should be moved to respective packages.

If config.CertFile and config.KeyFile are set when starting the broker, the broker will panic with the message "not implemented" when a client attempts to connect.

panic: not implemented

goroutine 7 [running]:
github.com/troian/easygo/netpoll.Must(0x0, 0x4933940, 0xc420202560, 0x4933940)
/Users/mochi/Development/Go/src/github.com/troian/easygo/netpoll/handle.go:46 +0x54
github.com/VolantMQ/volantmq/connection.(*impl).Accept(0xc42018a000, 0xd, 0xd, 0x493d380)
/Users/mochi/Development/Go/src/github.com/VolantMQ/volantmq/connection/connection.go:295 +0x9a
github.com/VolantMQ/volantmq/clients.(*Manager).Handle(0xc42024e1e0, 0x5909060, 0xc4200da4e0, 0x4933980, 0xc4201ffd80, 0x45a5240, 0xc42002df01)
/Users/mochi/Development/Go/src/github.com/VolantMQ/volantmq/clients/sessions.go:233 +0x388
github.com/VolantMQ/volantmq/transport.(*baseConfig).handleConnection(0xc420230000, 0x493e880, 0xc4200da4e0)
/Users/mochi/Development/Go/src/github.com/VolantMQ/volantmq/transport/base.go:85 +0x97
github.com/VolantMQ/volantmq/transport.(*tcp).Serve.func1(0xc420230000, 0x493e7c0, 0xc420680380)
/Users/mochi/Development/Go/src/github.com/VolantMQ/volantmq/transport/tcp.go:134 +0xf4
created by github.com/VolantMQ/volantmq/transport.(*tcp).Serve
/Users/mochi/Development/Go/src/github.com/VolantMQ/volantmq/transport/tcp.go:128 +0xd3
exit status 2

Draft is here

run paho testing: python3 client_test.py

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0xc86c89]

goroutine 13 [running]:
github.com/VolantMQ/volantmq/topics/mem.subscriptionRecurseSearch(0x0, 0xc420776f30, 0x1, 0x1, 0xc42000e168)
/home/bidong/.gvm/pkgsets/go1.9beta2/global/src/github.com/VolantMQ/volantmq/topics/mem/node.go:156 +0xa9
github.com/VolantMQ/volantmq/topics/mem.(*provider).subscriptionSearch(0xc420206480, 0xc420728ba0, 0x9, 0xc42000e168)
/home/bidong/.gvm/pkgsets/go1.9beta2/global/src/github.com/VolantMQ/volantmq/topics/mem/node.go:178 +0x120
github.com/VolantMQ/volantmq/topics/mem.(*provider).publisher(0xc420206480)
/home/bidong/.gvm/pkgsets/go1.9beta2/global/src/github.com/VolantMQ/volantmq/topics/mem/topics.go:255 +0x40a
created by github.com/VolantMQ/volantmq/topics/mem.NewMemProvider
/home/bidong/.gvm/pkgsets/go1.9beta2/global/src/github.com/VolantMQ/volantmq/topics/mem/topics.go:101 +0x966

Auth backend might be actually remote thus auth request might take significant time espessialy on publish/subscribe request due to permissions lookup and math.
Need some interface AuthCache provided as option to each auth provider.
Implementation should cache only successful auth requests. Form of storing might be like that:
username:
auth: allow
permissions: []array of successful permissions

Each cache entry might be invalidated

As per http://docs.oasis-open.org/mqtt/mqtt/v5.0/mqtt-v5.0.html

Go since 1.8 supports plugins (currently only Linux and Darwin).
To make core codebase small worth to move them out of repository

github.com\troian\easygo\netpoll\handle.go:97:38: cannot use desc.fd() (type int) as type syscall.Handle in argument to syscall.SetNonblock

In memory implementation should be as default and provided by persistence package. Other implementations should derive an API

.Subscribe failure test starting
Traceback (most recent call last):
File "client_test.py", line 361, in test_subscribe_failure
assert callback.subscribeds[0][1][0] == 0x80, "return code should be 0x80 %s" % callback.subscribeds
AssertionError: return code should be 0x80 [(14, [2])]

Logger need more configurable with interface type. So it will be more suitable for common use.

Although we also used zap-logger in production, but we still want to spec log level or something details for logger for debug or others.

Logger Level can be:

type Logger interface{
    Named(s string) Logger
    Debug(msg string, fields ...interface{})
    Info(msg string, fields ...interface{})
    Warn(msg string, fields ...interface{})
    Error(msg string, fields ...interface{}) 
    Fatal(msg string, fields ...interface{})
}

Make production logs more verbose.

• Connection logs

is there any way to get the publish message in server side without subscribe?
i want to save the publish message to db in volantmq server

https://github.com/golang/dep

Project needs split into couple repositories thus I created team VolantMQ for handling.

1. Rename project to VolantMQ (license pitfalls?)

routines.go use goring.Buffer
troian/goring only two commites,no goring.Buffer

Listen twice with differenct ConfigTCP&ConfigWS will do?

go srv.ListenAndServe(config)
srv.ListenAndServe(configWs)

Is this the correct way?

Currently active connection requires 4 running goroutines with 4K of stack per each plus incoming ring buffer of 8K. If scale to 1M active connections we get into 12G of memory. If client does send nothing and no publishes for subscribed topics it just waste of memory. Reasonable to optimize by packing into less goroutines and spin them only when events available.

1. Receiving packets. If there is no data on socket it blocked on epoll until there is some data. On github.com/golang/go, there is the issue. Wait until it's completed

2. Sending goroutine spin only when there is messages to transmit otherwise it should be down

Wish VolantMQ project to be a great MQTT project.

MQTT community need a great MQTT project of golang.

win7 64 bit ok

set goarch=386
win7 32 bit

{"level":"info","logger":"mqtt.example","caller":"surgemq/surgemq.go:76","msg":"Starting application"}
{"level":"info","logger":"mqtt.example","caller":"surgemq/surgemq.go:82","msg":"Initializing configs"}
{"level":"info","logger":"mqtt.example","caller":"surgemq/surgemq.go:115","msg":"Started listener","id":"tcp4://:1883"}
panic: runtime error: invalid memory address or nil pointer dereference
        panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xc0000005 code=0x0 addr=0x0 pc=0x40145b]

goroutine 7 [running]:
sync/atomic.StoreUint64(0x1146559c, 0x1, 0x0)
        c:/go/src/sync/atomic/asm_386.s:190 +0xb
github.com/troian/surgemq/buffer.(*Type).Close(0x11465560, 0x0, 0x0)
        d:/workspaces/go/sugermq/src/github.com/troian/surgemq/buffer/buffer.go:126 +0x3a
github.com/troian/surgemq/session.(*connection).stop(0x114980a0, 0x9f7a00)
        d:/workspaces/go/sugermq/src/github.com/troian/surgemq/session/connection.go:158 +0xbf
github.com/troian/surgemq/session.(*connection).onRoutineReturn(0x114980a0)
        d:/workspaces/go/sugermq/src/github.com/troian/surgemq/session/connection.go:315 +0x32
panic(0x7b0280, 0x9ee950)
        c:/go/src/runtime/panic.go:489 +0x22a
sync/atomic.LoadUint64(0x114655fc, 0xffffffff, 0x1)
        c:/go/src/sync/atomic/asm_386.s:159 +0xb
github.com/troian/surgemq/buffer.(*Type).isDone(0x114655c0, 0x114980d0)
        d:/workspaces/go/sugermq/src/github.com/troian/surgemq/buffer/buffer.go:630 +0x26
github.com/troian/surgemq/buffer.(*Type).WriteTo(0x114655c0, 0x11040a8, 0x114b9f70, 0x11040a8, 0x114b9f70, 0x7f1b01, 0x40a8d8)
        d:/workspaces/go/sugermq/src/github.com/troian/surgemq/buffer/buffer.go:195 +0x31
github.com/troian/surgemq/session.(*connection).sender(0x114980a0)
        d:/workspaces/go/sugermq/src/github.com/troian/surgemq/session/connection.go:304 +0x1e3
created by github.com/troian/surgemq/session.(*connection).start
        d:/workspaces/go/sugermq/src/github.com/troian/surgemq/session/connection.go:128 +0xd0

ref
https://my.oschina.net/u/115763/blog/198912

Auth interface provides API AclCheck. Add permissions check for Subscribe and Publish packets

1. If subscribe not allowed set retcode 0x80 for each respective topic does not meet permissions
2. If publish not allowed:
   • QoS 0 ignore
   • QoS 1 ignore but send PubAck
   • QoS 2 ignore but send PubRec

Hi,
Thanks for the hard work.
I'd like to use VolantMQ broker and I've followed the examples but when I try to connect to tcp within paho client I have the following error:

panic: send on closed channel

goroutine 39 [running]:
github.com/VolantMQ/volantmq/connection.(*impl).connectionRoutine(0xc420126000)
        /home/marc/go/src/github.com/VolantMQ/volantmq/connection/receiver.go:86 +0x156
github.com/VolantMQ/volantmq/connection.(*impl).rxConnection.func2(0xc420126000)
        /home/marc/go/src/github.com/VolantMQ/volantmq/connection/receiver.go:36 +0x2d
created by github.com/VolantMQ/volantmq/connection.(*impl).rxConnection
        /home/marc/go/src/github.com/VolantMQ/volantmq/connection/receiver.go:35 +0x78

Connecting to websocket I got:

panic: could not get file descriptor

goroutine 37 [running]:
github.com/troian/easygo/netpoll.Must(0x0, 0x80f260, 0xc420010ee0, 0x80f260)
        /home/marc/go/src/github.com/troian/easygo/netpoll/handle.go:46 +0x56
github.com/VolantMQ/volantmq/connection.(*impl).Accept(0xc4202e8000, 0xd, 0xd, 0x7fddbf716200)
        /home/marc/go/src/github.com/VolantMQ/volantmq/connection/connection.go:295 +0xa0
github.com/VolantMQ/volantmq/clients.(*Manager).Handle(0xc4202f20f0, 0x7fddc203c1f0, 0xc4200703c0, 0x7fddbf70e740, 0xc42000c140, 0x7fddbf4f7aa0, 0xc420027701)
        /home/marc/go/src/github.com/VolantMQ/volantmq/clients/sessions.go:233 +0x38c
github.com/VolantMQ/volantmq/transport.(*baseConfig).handleConnection(0xc420200000, 0x7fddbf717460, 0xc4200703c0)
        /home/marc/go/src/github.com/VolantMQ/volantmq/transport/base.go:85 +0x99
github.com/VolantMQ/volantmq/transport.(*ws).serveWs.func1(0xc420200000, 0xc4200f8280)
        /home/marc/go/src/github.com/VolantMQ/volantmq/transport/websocket.go:114 +0xf1
created by github.com/VolantMQ/volantmq/transport.(*ws).serveWs
        /home/marc/go/src/github.com/VolantMQ/volantmq/transport/websocket.go:109 +0x293

The client code I'm using (ws):

opts := pahoMQTT.NewClientOptions()
opts.AddBroker("ws://127.0.0.1:8080/")
opts.SetClientID("client-test")
opts.SetKeepAlive(time.Duration(120 * time.Second))
opts.SetConnectTimeout(time.Duration(5 * time.Second))
opts.SetAutoReconnect(false)

mqttClient := pahoMQTT.NewClient(opts)
token := mqttClient.Connect()
if token.WaitTimeout(time.Duration(5000*time.Millisecond)) || token.Error() != nil {
	log.Panic("Unable to connect to MQTT. ", token.Error())
}

Many thanks

i'm very like this project, do you have any roadmap plans for it? and have any plans to support cluster, websocket?