GithubHelp home page GithubHelp logo

wizzercn / mqttwk Goto Github PK

View Code? Open in Web Editor NEW
605.0 22.0 222.0 646 KB

Java + Netty 实现的高并发高可用MQTT服务broker，轻松支持10万并发(有群友实现了130万在线)

License: Apache License 2.0

Java 98.28% HTML 1.72%

mqttwk's Issues

单例模式通配符订阅消息重复发送

================ mqttwk =============
【发布】/topic1/1/1
【订阅收到】
/topic1/+/1 ：2条
/topic1/#：2条
================= moquette ============
【发布】/topic1/1/1
【订阅收到】
/topic1/+/1 ：1条
/topic1/#：1条

启动报错是什么原因

Plugin org.apache.maven.plugins:maven-install-plugin:2.4 or one of its dependencies could not be resolved: Failed to read artifact descriptor for org.apache.maven.plugins:maven-install-plugin:jar:2.4: 1 problem was encountered while building the effective model
[ERROR] [FATAL] Non-parseable POM D:\01_java_tools\repository\org\apache\maven\plugins\maven-install-plugin\2.4\maven-install-plugin-2.4.pom: unexpected character in markup (position: START_TAG seen ... function init() {\n loaded += 1;\n\n if (loaded < ... @44:24) @ line 44, column 24
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginResolutionException

代码不完整

[INFO] mqttwk 1.0-SNAPSHOT ................................ SUCCESS [ 0.284 s]
[INFO] wk-broker .......................................... FAILURE [ 6.043 s]
[INFO] wk-test ............................................ SKIPPED
[INFO] wk-test-device-client .............................. SKIPPED
[INFO] wk-test-server-client 1.0-SNAPSHOT ................. SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 7.064 s
[INFO] Finished at: 2018-07-03T15:52:29+08:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.3:compile (default-compile) on project wk-broker: Compilation failure: Compilation failure:
[ERROR] /source/third/MqttWk/wk-broker/src/main/java/cn/wizzer/mqttwk/mqtt/common/handler/MqttConnectHandler.java:[3,29] 找不到符号
[ERROR] 符号: 类 ConnectionDescriptor
[ERROR] 位置: 程序包 cn.wizzer.mqttwk.mqtt
[ERROR] /source/third/MqttWk/wk-broker/src/main/java/cn/wizzer/mqttwk/mqtt/common/handler/MqttConnectHandler.java:[4,29] 找不到符号
[ERROR] 符号: 类 ConnectionDescriptorStore
[ERROR] 位置: 程序包 cn.wizzer.mqttwk.mqtt
[ERROR] /source/third/MqttWk/wk-broker/src/main/java/cn/wizzer/mqttwk/mqtt/common/handler/MqttConnectHandler.java:[6,48] 找不到符号
[ERROR] 符号: 类 IConnectionsManager
[ERROR] 位置: 程序包 cn.wizzer.mqttwk.mqtt.common.connections
[ERROR] /source/third/MqttWk/wk-broker/src/main/java/cn/wizzer/mqttwk/mqtt/common/handler/MqttConnectHandler.java:[150,29] 找不到符号
[ERROR] 符号: 类 ConnectionDescriptor
[ERROR] 位置: 类 cn.wizzer.mqttwk.mqtt.common.handler.MqttConnectHandler
[ERROR] /source/third/MqttWk/wk-broker/src/main/java/cn/wizzer/mqttwk/mqtt/common/handler/MqttConnectHandler.java:[108,9] 找不到符号
[ERROR] 符号: 类 ConnectionDescriptor
[ERROR] 位置: 类 cn.wizzer.mqttwk.mqtt.common.handler.MqttConnectHandler
[ERROR] /source/third/MqttWk/wk-broker/src/main/java/cn/wizzer/mqttwk/mqtt/common/handler/MqttConnectHandler.java:[108,47] 找不到符号
[ERROR] 符号: 类 ConnectionDescriptor
[ERROR] 位置: 类 cn.wizzer.mqttwk.mqtt.common.handler.MqttConnectHandler
[ERROR] /source/third/MqttWk/wk-broker/src/main/java/cn/wizzer/mqttwk/mqtt/common/handler/MqttConnectHandler.java:[109,15] 找不到符号
[ERROR] 符号: 类 ConnectionDescriptor
[ERROR] 位置: 类 cn.wizzer.mqttwk.mqtt.common.handler.MqttConnectHandler
[ERROR] /source/third/MqttWk/wk-broker/src/main/java/cn/wizzer/mqttwk/mqtt/common/handler/MqttConnectHandler.java:[109,51] 找不到符号
[ERROR] 符号: 变量 connectionDescriptors
[ERROR] /source/third/MqttWk/wk-broker/src/main/java/cn/wizzer/mqttwk/mqtt/common/handler/MqttConnectHandler.java:[113,17] 找不到符号
[ERROR] 符号: 变量 connectionDescriptors
[ERROR] /source/third/MqttWk/wk-broker/src/main/java/cn/wizzer/mqttwk/mqtt/common/handler/MqttConnectHandler.java:[114,17] 找不到符号
[ERROR] 符号: 变量 connectionDescriptors
[ERROR] /source/third/MqttWk/wk-broker/src/main/java/cn/wizzer/mqttwk/mqtt/common/handler/MqttConnectHandler.java:[124,9] 找不到符号
[ERROR] 符号: 变量 m_interceptor
[ERROR] 位置: 类 cn.wizzer.mqttwk.mqtt.common.handler.MqttConnectHandler
[ERROR] /source/third/MqttWk/wk-broker/src/main/java/cn/wizzer/mqttwk/mqtt/common/handler/MqttConnectHandler.java:[126,37] 找不到符号
[ERROR] 符号: 变量 SENDACK
[ERROR] 位置: 类 cn.wizzer.mqttwk.mqtt.common.handler.MqttConnectHandler
[ERROR] /source/third/MqttWk/wk-broker/src/main/java/cn/wizzer/mqttwk/mqtt/common/handler/MqttConnectHandler.java:[126,46] 找不到符号
[ERROR] 符号: 变量 SESSION_CREATED
[ERROR] 位置: 类 cn.wizzer.mqttwk.mqtt.common.handler.MqttConnectHandler
[ERROR] /source/third/MqttWk/wk-broker/src/main/java/cn/wizzer/mqttwk/mqtt/common/handler/MqttConnectHandler.java:[130,15] 找不到符号
[ERROR] 符号: 类 ClientSession
[ERROR] 位置: 类 cn.wizzer.mqttwk.mqtt.common.handler.MqttConnectHandler
[ERROR] /source/third/MqttWk/wk-broker/src/main/java/cn/wizzer/mqttwk/mqtt/common/handler/MqttConnectHandler.java:[130,49] 找不到符号
[ERROR] 符号: 变量 sessionsRepository
[ERROR] /source/third/MqttWk/wk-broker/src/main/java/cn/wizzer/mqttwk/mqtt/common/handler/MqttConnectHandler.java:[138,26] 找不到符号
[ERROR] 符号: 变量 channel
[ERROR] 位置: 类 cn.wizzer.mqttwk.mqtt.common.handler.MqttConnectHandler
[ERROR] /source/third/MqttWk/wk-broker/src/main/java/cn/wizzer/mqttwk/mqtt/common/handler/MqttConnectHandler.java:[140,56] 找不到符号
[ERROR] 符号: 变量 MESSAGES_REPUBLISHED
[ERROR] 位置: 类 cn.wizzer.mqttwk.mqtt.common.handler.MqttConnectHandler
[ERROR] /source/third/MqttWk/wk-broker/src/main/java/cn/wizzer/mqttwk/mqtt/common/handler/MqttConnectHandler.java:[140,78] 找不到符号
[ERROR] 符号: 变量 ESTABLISHED
[ERROR] 位置: 类 cn.wizzer.mqttwk.mqtt.common.handler.MqttConnectHandler
[ERROR] /source/third/MqttWk/wk-broker/src/main/java/cn/wizzer/mqttwk/mqtt/common/handler/MqttConnectHandler.java:[152,56] 找不到符号
[ERROR] 符号: 变量 DISCONNECTED
[ERROR] 位置: 类 cn.wizzer.mqttwk.mqtt.common.handler.MqttConnectHandler
[ERROR] /source/third/MqttWk/wk-broker/src/main/java/cn/wizzer/mqttwk/mqtt/common/handler/MqttConnectHandler.java:[152,70] 找不到符号
[ERROR] 符号: 变量 SENDACK
[ERROR] 位置: 类 cn.wizzer.mqttwk.mqtt.common.handler.MqttConnectHandler
[ERROR] /source/third/MqttWk/wk-broker/src/main/java/cn/wizzer/mqttwk/mqtt/common/handler/MqttConnectHandler.java:[158,9] 找不到符号
[ERROR] 符号: 类 ClientSession
[ERROR] 位置: 类 cn.wizzer.mqttwk.mqtt.common.handler.MqttConnectHandler
[ERROR] /source/third/MqttWk/wk-broker/src/main/java/cn/wizzer/mqttwk/mqtt/common/handler/MqttConnectHandler.java:[158,43] 找不到符号
[ERROR] 符号: 变量 sessionsRepository
[ERROR] /source/third/MqttWk/wk-broker/src/main/java/cn/wizzer/mqttwk/mqtt/common/handler/MqttConnectHandler.java:[166,34] 找不到符号
[ERROR] 符号: 变量 mqttPacket
[ERROR] 位置: 类 cn.wizzer.mqttwk.mqtt.common.handler.MqttConnectHandler
[ERROR] /source/third/MqttWk/wk-broker/src/main/java/cn/wizzer/mqttwk/mqtt/common/handler/MqttConnectHandler.java:[171,21] 找不到符号
[ERROR] 符号: 变量 subscriptions
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn -rf :wk-broker

web客户端

大神好，MqttWk 没有web端的管理平台么？我看wiki里是下载mqtt-spy。

开启集群功能时会发送两条消息造成重复

开启集群后：
1）Publish -> processPublish -> internalCommunication.internalSend(internalMessage);
会广播一条消息，然后Broker各自收到消息判断是否在自己节点，然后下发消息。
Publish -> processPublish -> this.sendPublishMessage
正常发送消息。

如果Client1和Client2连接到同一个Broker，开启集群，Client1发送Client2订阅的主题，Client2除了集群收到一条，还会本地收到一条，产生两条信息造成重复。

解决：
InternalMessage增加BrokerId信息，判断是不是自己Broker
或者第二条send判断一下是否是集群模式，是就不发。

偶发的异常,未找到规律 

[DEBUG] 23:16:04.577 cn.wizzer.iot.mqtt.server.broker.protocol.Publish.lambda$sendPublishMessage$0(Publish.java:129) - PUBLISH - clientId: e16e6243ffd2fae1eb9b4c606bdf8b90, topic: /airm2m/tracker/862285033462874/will, Qos: 1, messageId: 2
[WARN ] 23:16:04.578 io.netty.util.internal.logging.Slf4JLogger.warn(Slf4JLogger.java:151) - An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
java.lang.IllegalArgumentException: messageId: 0 (expected: 1 ~ 65535)
        at io.netty.handler.codec.mqtt.MqttMessageIdVariableHeader.from(MqttMessageIdVariableHeader.java:31)
        at cn.wizzer.iot.mqtt.server.broker.protocol.Publish.sendPubAckMessage(Publish.java:161)
        at cn.wizzer.iot.mqtt.server.broker.protocol.Publish.processPublish(Publish.java:80)
        at cn.wizzer.iot.mqtt.server.broker.handler.BrokerHandler.userEventTriggered(BrokerHandler.java:132)
        at io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:329)
        at io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:315)
        at io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:307)
        at io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:108)
        at io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:353)
        at io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:329)
        at io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:315)
        at io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:307)
        at io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371)
        at io.netty.handler.timeout.IdleStateHandler$AllIdleTimeoutTask.run(IdleStateHandler.java:567)
        at io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:466)
        at io.netty.util.concurrent.PromiseTask$RunnableAdapter.call(PromiseTask.java:38)
        at io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:127)
        at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404)
        at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:322)
        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.lang.Thread.run(Thread.java:745)

你好，感谢。

我使用你的mqtt broker 代码，自己修改下，集群功能自己重新开发。

目前已上正式产线，集群3台，接入设备超20w，稳定运行半年多，目前非常稳定。

非常感谢。

请问是否，要在publish或其他操作的时候，同步redis的过期时间？

你好，向你请教一个问题：心跳
目前你是session信息存储到redis，
登录的时候，keepalive设置的是allIdleTime， channel.pipeline().addFirst("idle", new IdleStateHandler(0, 0, expire));
设备每次ping的时候， 延长redis的key失效时间。sessionStoreService.expire(clientId, sessionStore.getExpire());

假如：设备的那次心跳，服务器没有收到，redis的session缓存就会过期，消失。但是期间设备一直有publish操作，则userEventTriggered的idle事件不会触发。导致设备实际在线，在上层应用看来（从redis判断），则是一致离线。

请问是否，要再publish或其他操作的时候，同步redis的过期时间？

多机集群功能支持吗？

集群功能有实现吗？对于集群中三台机器A、B、C，现在：
设备d1连接机器A并订阅topic：/a/+/c，
设备d2连接机器B并订阅topic：/a/b/#，
设备d3连接机器C并pub消息到topic：/a/b/c，设备d1、设备d2能收到消息吗？

看源码pub消息时只是在本机取出了符合条件的订阅topic，从而找出对应的session，再找出对应的channel进行下发，但不在本机的会话就不能下发了。

这个channelIdMap并不是并发安全的吧

MqttWk/mqtt-broker/src/main/java/cn/wizzer/iot/mqtt/server/broker/handler/BrokerHandler.java

Lines 25 to 50 in e12930f

@IocBean
@ChannelHandler.Sharable
public class BrokerHandler extends SimpleChannelInboundHandler<MqttMessage> {
@Inject
private ProtocolProcess protocolProcess;
@Inject
private BrokerProperties brokerProperties;
@Inject
private ChannelGroup channelGroup;
@Inject
private Map<String, ChannelId> channelIdMap;
@Override
public void channelActive(ChannelHandlerContext ctx) throws Exception {
super.channelActive(ctx);
this.channelGroup.add(ctx.channel());
this.channelIdMap.put(brokerProperties.getId() + "_" + ctx.channel().id().asLongText(), ctx.channel().id());
}
@Override
public void channelInactive(ChannelHandlerContext ctx) throws Exception {
super.channelInactive(ctx);
this.channelGroup.remove(ctx.channel());
this.channelIdMap.remove(brokerProperties.getId() + "_" + ctx.channel().id().asLongText());
}

关于集群的几个问题

  1. 集群是主从模式的还是无中心化(decentralized)集群呢？
  2. 客户端session会在集群间复制吗？
    假设有三个n1,n2,n2，客户端c1。
    c1先连接n1节点（clean_session=false），然后断开连接，再连接n2节点，n2节点是否能够恢复该session？
  3. 如果两个相同clientId的客户端c1,c2。同时连接两个节点n1,n2。集群会保证踢掉其中一个客户端吗？
  4. 如果集群出现网络分区的问题，节点间如何恢复？
  5. 如果集群节点crash，有可能导致消息丢失吗？

谢谢

java.lang.NullPointerException报错

线上连接700台设备左右，频繁报空指针错误。有两个地方：
1、cn.wizzer.iot.mqtt.server.broker.protocol.DisConnect

    public void processDisConnect(Channel channel, MqttMessage msg) {
        String clientId = (String) channel.attr(AttributeKey.valueOf("clientId")).get();
        SessionStore sessionStore = sessionStoreService.get(clientId);
       // BUG：在极端条件下，redis中保存的session信息过期，导致sessionStore=null，此处报NullPointerException。
       // 判断条件修改为sessionStore != null && sessionStore.isCleanSession()
        if (sessionStore.isCleanSession()) {

        }
       // ...省略部分代码
    }

2、cn.wizzer.iot.mqtt.server.broker.handler.BrokerHandler

@Override
    public void userEventTriggered(ChannelHandlerContext ctx, Object evt) throws Exception {
       // ...省略部分代码
	// 发送遗嘱消息
	if (this.protocolProcess.getSessionStoreService().containsKey(clientId)) {
	    SessionStore sessionStore = this.protocolProcess.getSessionStoreService().get(clientId);
	    // BUG：IdleStateHandler检测间隔与redis超时时间相同。极端条件下，containsKey=true，但是get返回null。
            // 判断条件修改为：null != sessionStore && sessionStore.getWillMessage() != null
            // 或者考虑仅使用this.protocolProcess.getSessionStoreService().get(clientId)，然后判断null即可
	    if (sessionStore.getWillMessage() != null) {
	        this.protocolProcess.publish().processPublish(ctx.channel(), sessionStore.getWillMessage());
	    }
      // ...省略部分代码
    }

另外，BrokerHandler.exceptionCaught()方法最好能将异常信息打印出来，方便从日志中查看异常详细。以上问题就是加了打印才发现了具体的代码，如果不加仅打印了Exception的message：

    @Override
    public void exceptionCaught(ChannelHandlerContext ctx, Throwable cause) throws Exception {
        if (cause instanceof IOException) {
            // 远程主机强迫关闭了一个现有的连接的异常
            ctx.close();
        } else {
            // FIX：打印异常信息
            logger.error("出现了异常", cause);
            super.exceptionCaught(ctx, cause);
        }
    }

在使用org.eclipse.paho.client.mqttv3进行qos1级别消息发送时，接收客户端会出现由于会话过期而导致的链接断开

原因：在mqttv3客户端源码中，org.eclipse.paho.client.mqttv3.internal.ClientState#checkForActivity这个方法中会判断是否需要发送心跳。 其中有个变量lastInboundActivity会随着puback的发送而改变。这个变量会推迟心跳发送的时机。从而导致一个只接受消息而不发送消息的客户端在下次发送心跳的时间间隔会超过你设定的keepAlive*1.5的大小

建议在PubAck处理方法中添加对session的过期时间处理

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.