GithubHelp home page GithubHelp logo

volantmq / volantmq Goto Github PK

View Code? Open in Web Editor NEW
969.0 969.0 170.0 3.51 MB

High-Performance MQTT Server

License: Apache License 2.0

Go 97.31% Shell 0.45% Dockerfile 2.23%
broker iot iot-cloud mqtt mqtt-server websocket-transport

volantmq's Introduction

VolantMQ

CircleCI Codacy Badge codecov.io License

VolantMQ image *VolantMQ image by Marina Troian, licensed under [Creative Commons Attribution 4.0 International License][cc-by]

VolantMQ is a high performance MQTT broker that aims to be fully compliant with MQTT specs

##Features ###MQTT Specs

Network Transports

  • TCP
  • TLS
  • WebSocket
  • WebSocket+TLS

Persistence

By default server starts with in-memory persistence which means all sessions and messages lost after server restart.

Plugins

Auth

  • Server built-in basic auth.Key-value pairs in format user: sha256 of password provided by any of the following options

    • Users and their password hashes in config file
      - name: internal  # authenticator name, used by listeners
        backend: simpleAuth # authenticator type
        config:
          users:
            testuser: "9f735e0df9a1ddc702bf0a1a7b83033f9f7153a00c29de82cedadc9957289b05" # testpassword
    • Users and their password hashes in separate file
      - name: internal  # authenticator name, used by listeners
        backend: simpleAuth # authenticator type
        config:
          users: # both can be used simultaneously
            testuser: "9f735e0df9a1ddc702bf0a1a7b83033f9f7153a00c29de82cedadc9957289b05" # testpassword
          usersFile: <some path>
  • Build status HTTP

Monitoring

Persistence

  • In-Memory server built in
  • Build status BBolt

Debug

Health

Configuring

Server starts with default config from here. Any further configurations applied on top

Environment variables

  • VOLANTMQ_CONFIG - path to configuration file described in [this section](#Config file).
  • VOLANTMQ_PLUGIN_AUTH_HTTP_<NAME>_TOKEN - API token for auth plugins For example to supply auth token into auth plugin http1 from config below variable should be declared as VOLANTMQ_PLUGIN_AUTH_HTTP_HTTP1_TOKEN

Config file

File divided in a few sections Complete example can be found here

System

system:
  log:
    console:
      level: info # available levels: debug, info, warn, error, dpanic, panic, fatal
  http:
    defaultPort: 8080 # default HTTP listener assigned. Assigned to plugins like debug/health/metrics if they dont specify own port

Plugins

plugins:
    enabled: # list of plugins server will load on startup
      - systree
      - prometheus
      - debug
      - health
      - auth_http
      - persistence_bbolt
    config: # configuration of each plugin
      <plugin type>:
        - backed: systree # plugin name, allowed: systree, prometheus, http, prof.profiler, health, bbolt
          name: http1     # required by auth plugins only. Value used in auth.order
          config:         # configuration passed to plugin on load stage. refer to particular plugin for configuration

Default auth config

auth:
  order: # default auth order. Authenticators invoked in the order they present in the config. Listener can override
    - internal

MQTT specs

mqtt:
  version: // list of supported MQTT specifications
    - v3.1.1
    - v5.0
  keepAlive:
    period: 60 # KeepAlive The number of seconds to keep the connection live if there's no data.
    # Default is 60 seconds
    force: false # Force connection to use server keep alive interval (MQTT 5.0 only)
    # Default is false
  options:
    connectTimeout: 10 # The number of seconds to wait for the CONNACK message before disconnecting.
    # If not set then default to 2 seconds.
    offlineQoS0: true # OfflineQoS0 tell server to either persist (true) or not persist (false) QoS 0 messages for non-clean sessions
    # If not set than default is false
    sessionPreempt: true # AllowDuplicates Either allow or deny replacing of existing session if there new client with same clientID
    # If not set than default is false
    retainAvail: true # don't set to use default
    subsOverlap: false # tells server how to handle overlapping subscriptions from within one client
                       # - true server will send only one publish with max subscribed QoS even there are n subscriptions
                       # - false server will send as many publishes as amount of subscriptions matching publish topic exists
                       # Default is false
    subsId: false # don't set to use default
    subsShared: false # don't set to use default
    subsWildcard: true # don't set to use default
    receiveMax: 65535 # don't set to use default
    maxPacketSize: 268435455 # don't set to use default
    maxTopicAlias: 65535 # don't set to use default
    maxQoS: 2

Listeners

listeners:
  defaultAddr: "0.0.0.0" # default 127.0.0.1
  mqtt: # there are two types of listeners allowed tcp and ws (aka WebSocket) 
    tcp:
      1883:                # port number. can be as many ports configurations as needed
        host: 127.0.0.1    # optional. listen address. defaultAddr is used if omitted
        auth:              # optional. default auth configuration is used if omitted
          order:           # optional. default auth configuration is used if omitted
            - internal
      1884:
        auth:
          order:
            - http1
        tls:               # TLS configuration
          cert:            # path to certificate file
          key:             # path to key file
    ws:
      8883:
        path: mqtt
        auth:
          order:
            - http1
      8884:
        path: mqtt
        auth:
          order:
            - http1
        tls:               # TLS configuration
          cert:            # path to certificate file
          key:             # path to key file

Reason to have multiple listeners comes from performance impact of TLS as well as authentication Internal to system users can omit entire auth and TLS

   ┌──────────────┐                                
   │              │                                
   │ MQTT process │                                
   │              │                                
   └───────▲──────┘                                
           │                                       
           │             ╔════════════════════════╗
           │             ║ VolantMQ               ║
           │             ║                        ║
      ╔════▼═══╗         ║                        ║
      ║intranet◀═════════▶ 1883  # no auth, no TLS║
      ╚════════╝         ║                        ║
      ╔════════╗         ║                        ║
      ║internet◀═════════▶ 1884  # auth and TLS   ║
      ╚═▲══▲══▲╝         ║                        ║
        │  │  │          ╚════════════════════════╝
        │  │  │                                    
   ┌────┘  │  └───┐                                
   │       │      │                                
   │       │      │                                
   │       │      │                                
┌──▼─┐  ┌──▼─┐  ┌─▼──┐                             
│IoT1│  │IoT2│  │IoTn│                             
└────┘  └────┘  └────┘                             

Distribution

  • Docker image contains prebuilt plugins listed in this [section][#Plugins]
  • Helm

###How to use

docker run --rm -p 1883:1883 -p 8080:8080 -v $(pwd)/examples/config.yaml:/etc/volantmq/config.yaml \
--env VOLANTMQ_CONFIG=/etc/volantmq/config.yaml volantmq/volantmq

Credits

This project is heavily inspired by surgemq. As it was unmaintained it was forked to here, redesigned and packed with features since then

Appreciate JetBrains for granted license

volantmq's People

Contributors

aquarat avatar argylelabcoat avatar hades32 avatar muddydixon avatar myquant avatar scls19fr avatar take-cheeze avatar troian avatar vkorn avatar yqylovy avatar zhenjl avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

volantmq's Issues

panic: could not get file descriptor / panic: send on closed channel

Hi,
Thanks for the hard work.
I'd like to use VolantMQ broker and I've followed the examples but when I try to connect to tcp within paho client I have the following error:

panic: send on closed channel

goroutine 39 [running]:
github.com/VolantMQ/volantmq/connection.(*impl).connectionRoutine(0xc420126000)
        /home/marc/go/src/github.com/VolantMQ/volantmq/connection/receiver.go:86 +0x156
github.com/VolantMQ/volantmq/connection.(*impl).rxConnection.func2(0xc420126000)
        /home/marc/go/src/github.com/VolantMQ/volantmq/connection/receiver.go:36 +0x2d
created by github.com/VolantMQ/volantmq/connection.(*impl).rxConnection
        /home/marc/go/src/github.com/VolantMQ/volantmq/connection/receiver.go:35 +0x78

Connecting to websocket I got:

panic: could not get file descriptor

goroutine 37 [running]:
github.com/troian/easygo/netpoll.Must(0x0, 0x80f260, 0xc420010ee0, 0x80f260)
        /home/marc/go/src/github.com/troian/easygo/netpoll/handle.go:46 +0x56
github.com/VolantMQ/volantmq/connection.(*impl).Accept(0xc4202e8000, 0xd, 0xd, 0x7fddbf716200)
        /home/marc/go/src/github.com/VolantMQ/volantmq/connection/connection.go:295 +0xa0
github.com/VolantMQ/volantmq/clients.(*Manager).Handle(0xc4202f20f0, 0x7fddc203c1f0, 0xc4200703c0, 0x7fddbf70e740, 0xc42000c140, 0x7fddbf4f7aa0, 0xc420027701)
        /home/marc/go/src/github.com/VolantMQ/volantmq/clients/sessions.go:233 +0x38c
github.com/VolantMQ/volantmq/transport.(*baseConfig).handleConnection(0xc420200000, 0x7fddbf717460, 0xc4200703c0)
        /home/marc/go/src/github.com/VolantMQ/volantmq/transport/base.go:85 +0x99
github.com/VolantMQ/volantmq/transport.(*ws).serveWs.func1(0xc420200000, 0xc4200f8280)
        /home/marc/go/src/github.com/VolantMQ/volantmq/transport/websocket.go:114 +0xf1
created by github.com/VolantMQ/volantmq/transport.(*ws).serveWs
        /home/marc/go/src/github.com/VolantMQ/volantmq/transport/websocket.go:109 +0x293

The client code I'm using (ws):

opts := pahoMQTT.NewClientOptions()
opts.AddBroker("ws://127.0.0.1:8080/")
opts.SetClientID("client-test")
opts.SetKeepAlive(time.Duration(120 * time.Second))
opts.SetConnectTimeout(time.Duration(5 * time.Second))
opts.SetAutoReconnect(false)

mqttClient := pahoMQTT.NewClient(opts)
token := mqttClient.Connect()
if token.WaitTimeout(time.Duration(5000*time.Millisecond)) || token.Error() != nil {
	log.Panic("Unable to connect to MQTT. ", token.Error())
}

Many thanks

Make persistence as an interface

In memory implementation should be as default and provided by persistence package. Other implementations should derive an API

panic at disconnect

panic: sync: WaitGroup is reused before previous Wait has returned

goroutine 673928 [running]:
sync.(*WaitGroup).Wait(0xc425441328)
	/usr/local/Cellar/go/1.9/libexec/src/sync/waitgroup.go:133 +0xbb
github.com/VolantMQ/volantmq/connection.(*transmitter).shutdown.func1()
	/Users/amr/projects/gocode/src/github.com/VolantMQ/volantmq/connection/transmitter.go:67 +0x51
github.com/VolantMQ/volantmq/types.(*OnceWait).Do(0xc425441338, 0xc423859918, 0x13191e1)
	/Users/amr/projects/gocode/src/github.com/VolantMQ/volantmq/types/types.go:80 +0x77
github.com/VolantMQ/volantmq/connection.(*transmitter).shutdown(0xc4254412b0)
	/Users/amr/projects/gocode/src/github.com/VolantMQ/volantmq/connection/transmitter.go:64 +0x4e
github.com/VolantMQ/volantmq/connection.(*Type).onConnectionClose.func1()
	/Users/amr/projects/gocode/src/github.com/VolantMQ/volantmq/connection/netCallbacks.go:20 +0xb3
github.com/VolantMQ/volantmq/types.(*Once).Do(0xc421becb48, 0xc421831e08, 0xc4201621b0)
	/Users/amr/projects/gocode/src/github.com/VolantMQ/volantmq/types/types.go:108 +0x3d
github.com/VolantMQ/volantmq/connection.(*Type).onConnectionClose(0xc421becb00, 0x106f200, 0x0, 0x0)
	/Users/amr/projects/gocode/src/github.com/VolantMQ/volantmq/connection/netCallbacks.go:11 +0x5f
github.com/VolantMQ/volantmq/connection.(*Type).(github.com/VolantMQ/volantmq/connection.onConnectionClose)-fm(0xc424196700, 0x0, 0x0)
	/Users/amr/projects/gocode/src/github.com/VolantMQ/volantmq/connection/connection.go:175 +0x47
github.com/VolantMQ/volantmq/connection.(*receiver).routine.func1()
	/Users/amr/projects/gocode/src/github.com/VolantMQ/volantmq/connection/receiver.go:90 +0x7f
github.com/VolantMQ/volantmq/connection.(*receiver).routine(0xc4241966c0, 0x1)
	/Users/amr/projects/gocode/src/github.com/VolantMQ/volantmq/connection/receiver.go:95 +0x3b9
created by github.com/VolantMQ/volantmq/connection.(*receiver).run
	/Users/amr/projects/gocode/src/github.com/VolantMQ/volantmq/connection/receiver.go:78 +0xc2

build fail on windows

github.com\troian\easygo\netpoll\handle.go:97:38: cannot use desc.fd() (type int) as type syscall.Handle in argument to syscall.SetNonblock

Make persistence as plugins

Go since 1.8 supports plugins (currently only Linux and Darwin).
To make core codebase small worth to move them out of repository

goring forget push?

routines.go use goring.Buffer
troian/goring only two commites,no goring.Buffer

panic during paho.mqtt.testing

run paho testing: python3 client_test.py

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0xc86c89]

goroutine 13 [running]:
github.com/VolantMQ/volantmq/topics/mem.subscriptionRecurseSearch(0x0, 0xc420776f30, 0x1, 0x1, 0xc42000e168)
/home/bidong/.gvm/pkgsets/go1.9beta2/global/src/github.com/VolantMQ/volantmq/topics/mem/node.go:156 +0xa9
github.com/VolantMQ/volantmq/topics/mem.(*provider).subscriptionSearch(0xc420206480, 0xc420728ba0, 0x9, 0xc42000e168)
/home/bidong/.gvm/pkgsets/go1.9beta2/global/src/github.com/VolantMQ/volantmq/topics/mem/node.go:178 +0x120
github.com/VolantMQ/volantmq/topics/mem.(*provider).publisher(0xc420206480)
/home/bidong/.gvm/pkgsets/go1.9beta2/global/src/github.com/VolantMQ/volantmq/topics/mem/topics.go:255 +0x40a
created by github.com/VolantMQ/volantmq/topics/mem.NewMemProvider
/home/bidong/.gvm/pkgsets/go1.9beta2/global/src/github.com/VolantMQ/volantmq/topics/mem/topics.go:101 +0x966

Middleware option

Is it possible to add some middleware option like in the HTTP package? For our project I need to process incoming messages before sending it to the subscribers.

Implement permissions check

Auth interface provides API AclCheck. Add permissions check for Subscribe and Publish packets

  1. If subscribe not allowed set retcode 0x80 for each respective topic does not meet permissions
  2. If publish not allowed:
    • QoS 0 ignore
    • QoS 1 ignore but send PubAck
    • QoS 2 ignore but send PubRec

add empty Persistence provider

don't want persistence anything
or if mqtt client library bug,then don't reply such as PUBCOMP,broker memory will leak?

so want a empty Persistence provider

"Panic: not implemented"; when client connects using CertFile and KeyFile.

If config.CertFile and config.KeyFile are set when starting the broker, the broker will panic with the message "not implemented" when a client attempts to connect.

panic: not implemented

goroutine 7 [running]:
github.com/troian/easygo/netpoll.Must(0x0, 0x4933940, 0xc420202560, 0x4933940)
/Users/mochi/Development/Go/src/github.com/troian/easygo/netpoll/handle.go:46 +0x54
github.com/VolantMQ/volantmq/connection.(*impl).Accept(0xc42018a000, 0xd, 0xd, 0x493d380)
/Users/mochi/Development/Go/src/github.com/VolantMQ/volantmq/connection/connection.go:295 +0x9a
github.com/VolantMQ/volantmq/clients.(*Manager).Handle(0xc42024e1e0, 0x5909060, 0xc4200da4e0, 0x4933980, 0xc4201ffd80, 0x45a5240, 0xc42002df01)
/Users/mochi/Development/Go/src/github.com/VolantMQ/volantmq/clients/sessions.go:233 +0x388
github.com/VolantMQ/volantmq/transport.(*baseConfig).handleConnection(0xc420230000, 0x493e880, 0xc4200da4e0)
/Users/mochi/Development/Go/src/github.com/VolantMQ/volantmq/transport/base.go:85 +0x97
github.com/VolantMQ/volantmq/transport.(*tcp).Serve.func1(0xc420230000, 0x493e7c0, 0xc420680380)
/Users/mochi/Development/Go/src/github.com/VolantMQ/volantmq/transport/tcp.go:134 +0xf4
created by github.com/VolantMQ/volantmq/transport.(*tcp).Serve
/Users/mochi/Development/Go/src/github.com/VolantMQ/volantmq/transport/tcp.go:128 +0xd3
exit status 2

Implement auth and permissions caching

Auth backend might be actually remote thus auth request might take significant time espessialy on publish/subscribe request due to permissions lookup and math.
Need some interface AuthCache provided as option to each auth provider.
Implementation should cache only successful auth requests. Form of storing might be like that:
username:
auth: allow
permissions: []array of successful permissions

Each cache entry might be invalidated

connection.go:217 opanic: file tcp 127.0.0.1:1883->127.0.0.1:1691: not supported by windows(VolantMQ/volantmq/connection/connection.go:217 )

func (s *Type) Start() {
	s.onStart.Do(func() {
		s.txRun()
		s.EventPoll.Start(s.Desc, s.rxRun) // nolint: errcheck
		s.started.Done()
	})
}

panic: file tcp 127.0.0.1:1883->127.0.0.1:1691: not supported by windows
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xc0000005 code=0x0 addr=0x0 pc=0x710bca]

goroutine 33 [running]:
github.com/VolantMQ/volantmq/connection.(*Type).Start(0x0)
E:/LzGo/volantmq/src/github.com/VolantMQ/volantmq/connection/connection.go:217 +0x3a
github.com/VolantMQ/volantmq/clients.(*session).start(0xc04216e000)
E:/LzGo/volantmq/src/github.com/VolantMQ/volantmq/clients/session.go:142 +0x93
github.com/VolantMQ/volantmq/clients.(*Manager).NewSession.func1(0xc042177b40, 0xc04223dc48, 0xc0420ea000, 0xc042177b30, 0xc042177b28, 0xc042177b20)
E:/LzGo/volantmq/src/github.com/VolantMQ/volantmq/clients/sessions.go:160 +0x4b7
panic(0x891840, 0xc04265a0f0)
E:/Go/src/runtime/panic.go:491 +0x291
github.com/troian/easygo/netpoll.Must(0x0, 0xb5d840, 0xc04265a0f0, 0xb5d840)
E:/LzGo/volantmq/src/github.com/troian/easygo/netpoll/handle.go:41 +0x5b
github.com/VolantMQ/volantmq/clients.(*Manager).newConnectionPreConfig(0xc0420ea000, 0xc042177c48, 0x0)
E:/LzGo/volantmq/src/github.com/VolantMQ/volantmq/clients/sessions.go:299 +0x99
github.com/VolantMQ/volantmq/clients.(*Manager).configureSession(0xc0420ea000, 0xc042177c48, 0xc04216e000, 0xc042088060, 0x20, 0xc04216e000, 0x0, 0x0, 0x5594a0294570c68c)
E:/LzGo/volantmq/src/github.com/VolantMQ/volantmq/clients/sessions.go:322 +0x14c
github.com/VolantMQ/volantmq/clients.(*Manager).NewSession(0xc0420ea000, 0xc04223dc48)
E:/LzGo/volantmq/src/github.com/VolantMQ/volantmq/clients/sessions.go:181 +0x22c
github.com/VolantMQ/volantmq/transport.(*baseConfig).handleConnection(0xc042042000, 0xb65f20, 0xc04208a040)
E:/LzGo/volantmq/src/github.com/VolantMQ/volantmq/transport/base.go:147 +0xb2d
github.com/VolantMQ/volantmq/transport.(*tcp).Serve.func1(0xc042042000, 0xb660a0, 0xc042162000)
E:/LzGo/volantmq/src/github.com/VolantMQ/volantmq/transport/tcp.go:136 +0xfb
created by github.com/VolantMQ/volantmq/transport.(*tcp).Serve
E:/LzGo/volantmq/src/github.com/VolantMQ/volantmq/transport/tcp.go:130 +0xda

Session panics when encountering errors

When any error occurs in clients/session.go, the library panics because the logger hasn't been instantiated.

goroutine 21 [running]:
go.uber.org/zap.(*Logger).check(0x0, 0x2, 0x463d310, 0x1f, 0xc42105f4c0)
/Users/mochi/Development/Go/src/go.uber.org/zap/logger.go:266 +0xfd
go.uber.org/zap.(*Logger).Error(0x0, 0x463d310, 0x1f, 0xc42105f4c0, 0x1, 0x1)
/Users/mochi/Development/Go/src/go.uber.org/zap/logger.go:202 +0x44
github.com/VolantMQ/volantmq/clients.(*session).SignalUnSubscribe(0xc4200a4000, 0xc4212b87e0, 0xc420febcd7, 0xc420018432, 0x1, 0xc420febd50)
/Users/mochi/Development/Go/src/github.com/VolantMQ/volantmq/clients/session.go:234 +0x34e
github.com/VolantMQ/volantmq/connection.(*impl).processIncoming(0xc420098500, 0x4945760, 0xc4212b87e0, 0xc4212b87e0, 0x0)
/Users/mochi/Development/Go/src/github.com/VolantMQ/volantmq/connection/connection.go:633 +0x437
github.com/VolantMQ/volantmq/connection.(*impl).rxRoutine(0xc420098500)
/Users/mochi/Development/Go/src/github.com/VolantMQ/volantmq/connection/receiver.go:59 +0x24b
github.com/VolantMQ/volantmq/connection.(*impl).rxRun.func1(0xc420098500)
/Users/mochi/Development/Go/src/github.com/VolantMQ/volantmq/connection/receiver.go:22 +0x5e
created by github.com/VolantMQ/volantmq/connection.(*impl).rxRun
/Users/mochi/Development/Go/src/github.com/VolantMQ/volantmq/connection/receiver.go:19 +0x9d
exit status 2

It appears that when a newSession is created from the session manager, no instance of Zap logger is transferred from the manager.

The problem can be resolved by adding ses.log = m.log to sessions.go:434, however since I don't know enough about the current and planned architecture for VolantMQ I haven't issued a request. Maybe @troian can advise?

(fantastic work on all this, by the way).

func (m *Manager) allocContainer(id string, createdAt time.Time, cn connection.Session) *container {
ses := newSession(sessionPreConfig{
id: id,
createdAt: createdAt,
conn: cn,
messenger: m.TopicsMgr,
persistence: m.persistence,
})

wrap := &container{}
ses.idLock = &wrap.lock
ses.log = m.log // <-- :434 transfer manager logger to session
wrap.ses.Store(ses)
wrap.acquire()

return wrap
}

Logger should be interface type

Logger need more configurable with interface type. So it will be more suitable for common use.

Although we also used zap-logger in production, but we still want to spec log level or something details for logger for debug or others.

Logger Level can be:

type Logger interface{
    Named(s string) Logger
    Debug(msg string, fields ...interface{})
    Info(msg string, fields ...interface{})
    Warn(msg string, fields ...interface{})
    Error(msg string, fields ...interface{}) 
    Fatal(msg string, fields ...interface{})
}

Transfer project to VolantMQ

Project needs split into couple repositories thus I created team VolantMQ for handling.

  1. Rename project to VolantMQ (license pitfalls?)

win32 support

win7 64 bit ok

set goarch=386
win7 32 bit

{"level":"info","logger":"mqtt.example","caller":"surgemq/surgemq.go:76","msg":"Starting application"}
{"level":"info","logger":"mqtt.example","caller":"surgemq/surgemq.go:82","msg":"Initializing configs"}
{"level":"info","logger":"mqtt.example","caller":"surgemq/surgemq.go:115","msg":"Started listener","id":"tcp4://:1883"}
panic: runtime error: invalid memory address or nil pointer dereference
        panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xc0000005 code=0x0 addr=0x0 pc=0x40145b]

goroutine 7 [running]:
sync/atomic.StoreUint64(0x1146559c, 0x1, 0x0)
        c:/go/src/sync/atomic/asm_386.s:190 +0xb
github.com/troian/surgemq/buffer.(*Type).Close(0x11465560, 0x0, 0x0)
        d:/workspaces/go/sugermq/src/github.com/troian/surgemq/buffer/buffer.go:126 +0x3a
github.com/troian/surgemq/session.(*connection).stop(0x114980a0, 0x9f7a00)
        d:/workspaces/go/sugermq/src/github.com/troian/surgemq/session/connection.go:158 +0xbf
github.com/troian/surgemq/session.(*connection).onRoutineReturn(0x114980a0)
        d:/workspaces/go/sugermq/src/github.com/troian/surgemq/session/connection.go:315 +0x32
panic(0x7b0280, 0x9ee950)
        c:/go/src/runtime/panic.go:489 +0x22a
sync/atomic.LoadUint64(0x114655fc, 0xffffffff, 0x1)
        c:/go/src/sync/atomic/asm_386.s:159 +0xb
github.com/troian/surgemq/buffer.(*Type).isDone(0x114655c0, 0x114980d0)
        d:/workspaces/go/sugermq/src/github.com/troian/surgemq/buffer/buffer.go:630 +0x26
github.com/troian/surgemq/buffer.(*Type).WriteTo(0x114655c0, 0x11040a8, 0x114b9f70, 0x11040a8, 0x114b9f70, 0x7f1b01, 0x40a8d8)
        d:/workspaces/go/sugermq/src/github.com/troian/surgemq/buffer/buffer.go:195 +0x31
github.com/troian/surgemq/session.(*connection).sender(0x114980a0)
        d:/workspaces/go/sugermq/src/github.com/troian/surgemq/session/connection.go:304 +0x1e3
created by github.com/troian/surgemq/session.(*connection).start
        d:/workspaces/go/sugermq/src/github.com/troian/surgemq/session/connection.go:128 +0xd0

ref
https://my.oschina.net/u/115763/blog/198912

Optimize goroutine and memory usage

Currently active connection requires 4 running goroutines with 4K of stack per each plus incoming ring buffer of 8K. If scale to 1M active connections we get into 12G of memory. If client does send nothing and no publishes for subscribed topics it just waste of memory. Reasonable to optimize by packing into less goroutines and spin them only when events available.

  1. Receiving packets. If there is no data on socket it blocked on epoll until there is some data. On github.com/golang/go, there is the issue. Wait until it's completed

  2. Sending goroutine spin only when there is messages to transmit otherwise it should be down

paho.mqtt.testing fails

.Subscribe failure test starting
Traceback (most recent call last):
File "client_test.py", line 361, in test_subscribe_failure
assert callback.subscribeds[0][1][0] == 0x80, "return code should be 0x80 %s" % callback.subscribeds
AssertionError: return code should be 0x80 [(14, [2])]

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.