GithubHelp home page GithubHelp logo

rocketmq-mqtt's Introduction

Apache RocketMQ MQTT

Build Status License CodeCov Average time to resolve an issue Percentage of issues still open Twitter Follow

A new MQTT protocol architecture model, based on which RocketMQ can better support messages from terminals such as IoT devices and Mobile APP. Based on the RocketMQ message unified storage engine, it supports both MQTT terminal and server message sending and receiving.The entire project design refer to MQTT overview.

Architecture

The relevant architecture design is introduced in RIP-33. The queue model of MQTT needs to be based on the light message queue, For the design of the specific light message queue, please refer to the RIP-28(LMQ)-wiki of RocketMQ.

Get Started

Prerequisites

The queue model of MQTT needs to be based on the light message queue feature (RIP-28) of RocketMQ. RocketMQ has only supported this feature since version 4.9.3. Please ensure that the installed version of RocketMQ already supports this feature.

For the quick start of light message queue, please refer to Example_LMQ of RocketMQ. For example, set the following parameters to true in broker.conf

enableLmq = true
enableMultiDispatch = true

Build Requirements

The current project requires JDK 1.8.x. When building on MAC arm64 the recommended JDK 1.8 must be based on 386 architecture or use Maven flag -Dos.arch=x86_64 when building with Maven.

  1. Clone
git clone https://github.com/apache/rocketmq-mqtt
  1. Build the package
cd rocketmq-mqtt
mvn -Prelease-all -DskipTests clean install -U 
  1. Config
cd distribution/target/
cd conf

Some important configuration items in the service.conf configuration file

Config Key Instruction
username used for auth
secretKey used for auth
NAMESRV_ADDR specify namesrv address
eventNotifyRetryTopic notify event retry topic
clientRetryTopic client retry topic
metaAddr meta all nodes ip:port. Same as membersAddress in meta.config

And some configuration items in the meta.conf configuration file

Config Key Instruction
selfAddress meta cur node ip:port, e.g. 192.168.0.1:8080
membersAddress meta all nodes ip:port, e.g. 192.168.0.1:8080,192.168.0.2:8080,192.168.0.3:8080
  1. CreateTopic

    create all first-level topics, including eventNotifyRetryTopic and clientRetryTopic in the configuration file above.

sh mqadmin updatetopic -c {cluster} -t {topic} -n {namesrv}
  1. Initialize Meta
  • Configure Gateway Node List
sh mqadmin updateKvConfig -s LMQ -k LMQ_CONNECT_NODES -v {ip1,ip2} -n {namesrv}
  • Configure the first-level topic list
sh mqadmin updateKvConfig -s LMQ -k ALL_FIRST_TOPICS -v {topic1,topic2} -n {namesrv}
  • Configure a list of wildcard characters under each first-level topic
sh mqadmin updateKvConfig  -s LMQ -k {topic} -v {topic/+}  -n {namesrv}
  1. Start Process
cd bin
sh meta.sh start
sh mqtt.sh start

Example

The mqtt-example module has written basic usage example code, which can be used for reference

Protocol Version

The currently supported protocol version is MQTT 3.1.1.

Authentication

At present, an implementation based on the HmacSHA1 signature algorithm is provided by default, Refer to AuthManagerSample. Users can customize other implementations to meet the needs of businesses to flexibly verify resources and identities.

Meta Persistence

At present, meta data storage and management is simply implemented through the kvconfig mechanism of namesrv by default, Refer to MetaPersistManagerSample. Users can customize other implementations.

License

Apache License, Version 2.0 Copyright (C) Apache Software Foundation.

rocketmq-mqtt's People

Contributors

aliensb avatar changingfond avatar coldluna avatar dependabot[bot] avatar dongyuanpan avatar duhenglucky avatar ferrirw avatar fujian-zfj avatar hill007299 avatar jlleitschuh avatar meshpaul avatar oliverwqcwrw avatar pingww avatar rongtongjin avatar scutcr7 avatar shannonding avatar shendongsd avatar tianliuliu avatar wangzhiwei1314 avatar yxac avatar yyyywd avatar zhouxinyu avatar zongtanghu avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

rocketmq-mqtt's Issues

Practice sharing

We use the same approach for communication between applications and services
but not by a gateway
we use a mqtt client to send message from rocketmq to mqtt broker
by a gateway seem to reduce number of communication between mqtt-client and mqtt-broker

Upgrade logback-classic version

I noticed this pr,github dependabot bumped logback-core version from 1.1.5 to 1.2.9, but logback-classic version is still 1.1.5. Thus, when i tried to run project, i met the error below.

Failed to instantiate [ch.qos.logback.classic.LoggerContext]
Reported exception:
java.lang.NoSuchMethodError: ch.qos.logback.core.util.Loader.getResourceOccurrenceCount(Ljava/lang/String;Ljava/lang/ClassLoader;)Ljava/util/Set;
	at ch.qos.logback.classic.util.ContextInitializer.multiplicityWarning(ContextInitializer.java:173)
	at ch.qos.logback.classic.util.ContextInitializer.statusOnResourceSearch(ContextInitializer.java:196)
	at ch.qos.logback.classic.util.ContextInitializer.getResource(ContextInitializer.java:143)
	at ch.qos.logback.classic.util.ContextInitializer.findURLOfDefaultConfigurationFile(ContextInitializer.java:137)
	at ch.qos.logback.classic.util.ContextInitializer.autoConfig(ContextInitializer.java:150)
	at org.slf4j.impl.StaticLoggerBinder.init(StaticLoggerBinder.java:85)
	at org.slf4j.impl.StaticLoggerBinder.<clinit>(StaticLoggerBinder.java:55)
	at org.slf4j.LoggerFactory.bind(LoggerFactory.java:143)
	at org.slf4j.LoggerFactory.performInitialization(LoggerFactory.java:122)
	at org.slf4j.LoggerFactory.getILoggerFactory(LoggerFactory.java:378)
	at org.apache.rocketmq.logging.Slf4jLoggerFactory.<init>(Slf4jLoggerFactory.java:26)
	at org.apache.rocketmq.logging.InternalLoggerFactory.<clinit>(InternalLoggerFactory.java:65)
	at org.apache.rocketmq.common.MixAll.<clinit>(MixAll.java:88)
	at org.apache.rocketmq.mqtt.ds.config.ServiceConf.<init>(ServiceConf.java:52)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:142)
	at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:89)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateBean(AbstractAutowireCapableBeanFactory.java:1151)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1103)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:513)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483)
	at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:312)
	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
	at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:308)
	at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:761)
	at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:867)
	at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:543)
	at org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:139)
	at org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:83)
	at org.apache.rocketmq.mqtt.cs.starter.Startup.main(Startup.java:30)

Are you using RocketMQ MQTT?

Are you using RocketMQ MQTT?

As the parts of the next generation of the Apache RocketMQ. The project can better support messages from terminals such as IoT devices and mobile apps with the advent of the IoT. Today, we sincerely invite you to take a minute to feedback on your usage scenario, whether you pass here by chance, or you are a frequent user in the IoT or offering Communications Platform as a Service (cPaaS).

The purpose of this issue

We are always interested in finding out who is using our MQTT solution, what attracted you to use it, how we can listen to your needs, and if you are interested, help promote your organization.

What we would like from you

Pls. submit a comment in this issue to include the following information:

  • your company, school, or organization
  • your country and city
  • your contact info, such as email, WeChat, and Twitter (optional).
  • usage scenario
  • expectations(optional)

You can refer to the following sample answer for the format:

* Organization: XX Company
* Location: Seoul, South Korea
* Contact: [email protected]
* Version: v1.0.0
* Status: production
* Expectations(optional): Integrated with HIveMQ, support clustering and metadata management. 

Thanks again for your participation!
Apache RocketMQ Community

Confusing why append '/' to the end of topic name?

Have not find related explain, please give some description.

/**
* t1/t2/t3/
*
* @param topic
* @return
*/
public static String normalizeTopic(String topic) {
if (topic == null) {
return null;
}
if (!topic.contains(Constants.MQTT_TOPIC_DELIMITER)) {
return topic;
}
if (!topic.endsWith(Constants.MQTT_TOPIC_DELIMITER)) {
return topic + Constants.MQTT_TOPIC_DELIMITER;
}
return topic;
}

To support Retain Message Feature

Retain Message are also a feature of the MQTT 3.1.1 protocol that is not supported in our current release and we plan to discuss implementation solutions for this protocol feature.

Add a lightweight, embedded distributed KV database

add a distributed KV database is necessary, we can use it to implement some important features.

  1. used to store and manage metadata, such as first topic, wildcard management, subscription relationships, etc.
  2. clientid conflict detection. clientId conflict detection has not yet been implemented, because there is no distributed metadata center, only single-machine conflict detection can be achieved.
  3. lookup. The lookup component is mainly used to optimize event broadcasting, repeated connection and mutual kicking, and online status query. It is not necessary for the core link of message sending and receiving. The lookup component is actually a KV query. Of course, you can use redis directly, or you can consider the built-in implementation of a simple of memory KV.
  4. will and retain can be implement by distributed kv convenient。

Some interface have no comment

in the branch main
the interface org.apache.rocketmq.mqtt.ds.upstream.UpstreamProcessor#process
method have no comment
image

Concurrent unit test failure problem

Tests run: 3, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 2.025 sec <<< FAILURE!
testDoHandlerSuccess(org.apache.rocketmq.mqtt.cs.test.protocol.mqtt.handler.TestMqttConnectHandler) Time elapsed: 1.018 sec <<< FAILURE!
Wanted but not invoked:
channel.writeAndFlush(

);
-> at org.apache.rocketmq.mqtt.cs.test.protocol.mqtt.handler.TestMqttConnectHandler.testDoHandlerSuccess(TestMqttConnectHandler.java:131)
However, there were exactly 5 interactions with this mock:
channel.attr(I);
-> at org.apache.rocketmq.mqtt.cs.channel.ChannelInfo.getInfo(ChannelInfo.java:245)
channel.attr(I);
-> at org.apache.rocketmq.mqtt.cs.channel.ChannelInfo.getInfo(ChannelInfo.java:245)
channel.attr(I);
-> at org.apache.rocketmq.mqtt.cs.channel.ChannelInfo.getInfo(ChannelInfo.java:245)
channel.attr(I);
-> at org.apache.rocketmq.mqtt.cs.channel.ChannelInfo.getInfo(ChannelInfo.java:245)
channel.attr(I);
-> at org.apache.rocketmq.mqtt.cs.channel.ChannelInfo.getInfo(ChannelInfo.java:245)
at org.apache.rocketmq.mqtt.cs.test.protocol.mqtt.handler.TestMqttConnectHandler.testDoHandlerSuccess(TestMqttConnectHandler.java:131)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.mockito.internal.runners.DefaultInternalRunner$1$1.evaluate(DefaultInternalRunner.java:44)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.mockito.internal.runners.DefaultInternalRunner$1.run(DefaultInternalRunner.java:74)
at org.mockito.internal.runners.DefaultInternalRunner.run(DefaultInternalRunner.java:80)
at org.mockito.internal.runners.StrictRunner.run(StrictRunner.java:39)
at org.mockito.junit.MockitoJUnitRunner.run(MockitoJUnitRunner.java:163)
at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252)
at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141)
at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
at org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115)
at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75)

Meet NoSuchBeanDefinitionException when subscribe topic

PR 79 defined a subscription storage interface, but there is no default implementation, then SpringUtils.getBean(SubscriptionPersistManager.class) will throw a NoSuchBeanDefinitionException, which is a runtime exception, client's subscribe/unsubscribe operations will be failed.

Solution: Catch BeansException in SpringUtils class and return null, to not block normal logic.

image

2022-05-20 21:40:21:072 [AuthHook_2] ERROR [AbstractUpstreamHook] 
org.springframework.beans.factory.NoSuchBeanDefinitionException: No qualifying bean of type 'org.apache.rocketmq.mqtt.common.facade.SubscriptionPersistManager' available
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBean(DefaultListableBeanFactory.java:353)
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBean(DefaultListableBeanFactory.java:340)
	at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1092)
	at org.apache.rocketmq.mqtt.common.util.SpringUtils.getBean(SpringUtils.java:35)
	at org.apache.rocketmq.mqtt.ds.upstream.processor.SubscribeProcessor.process(SubscribeProcessor.java:65)
	at org.apache.rocketmq.mqtt.ds.upstream.UpstreamProcessorManager.processMqttMessage(UpstreamProcessorManager.java:74)
	at org.apache.rocketmq.mqtt.common.hook.AbstractUpstreamHook.doHook(AbstractUpstreamHook.java:44)
	at org.apache.rocketmq.mqtt.common.hook.AbstractUpstreamHook.lambda$doHook$0(AbstractUpstreamHook.java:54)

Some code can be Optimize

there are some code can be Optimized in main branch

image

such as the queueName variable and can use if else in the condition "TopicUtils.isP2P(secondTopic)"

check upstreamHookResult before enter message handler main logic

image

Take MqttConnectHandler for example,as show in the above graph. While check upstream hook result return false in code block 2, schedule.schedule() method in code block 1 is unnecessary to run. So, moving code block 2 in front of 1 can reduce run time, maybe just a little, but it is according with what "upstream hook" means.

Add PreHandler For Preprocessing

On the upstream processing link, there needs to be a preprocessing logic for validity check, such as deduplication processing of qos2

To support Will Message Feature

Will Message is MQTT's ability to gracefully send wills to third parties for devices that may be unexpectedly disconnected. The Current version supports the MQTT 3.1.1 protocol, but the Will Message feature in this version is not yet supported. So we are discussing plans to support this feature.

Fix the bug of Get Start in Readme

cp -r  target/rocketmq-mqtt ~
cd ~/rocketmq-mqtt
cd conf

should be

cp -r  target/rocketmq-mqtt/rocketmq-mqtt ~
cd ~/rocketmq-mqtt
cd conf

Support Shared Subscriptions feature

The current version seems does not support shared subscriptions, I am interested in this and willing to contribute it for some time in future,may I?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.