didi / ddmq

DDMQ is a distributed messaging product with low latency, high throughput and high availability.

English | 中文

DDMQ is a distributed messaging product built by DiDi Infrastructure Team based on Apache RocketMQ. As a distributed messaging middleware, DDMQ provides low latency, high throughput and high available messaging service to many important large-scale distributed systems inside DiDi. DDMQ provides realtime messaging, delay-time messaging and transactional messaging to satisfy different scenarios. Through an easy-to-use Web Console and simple SDK Client, developers can experience producing and consuming messages in the most simple and stable way.

---

Features

• Messaging model: support both P2P and Pub/Sub messaging model

• Massive message storage, support both RocketMQ and Kafka as storage engine

• Low latency & High throughput

• Delay message, use RocksDB as storage engine

• Transactional message: provide transaction similar to X/Open XA

• Multiple language client SDK: provide client SDK in PHP, Java, Go, C/C++, Python

• Message transition and filter with user-defined Groovy script

• An easy-to-use Web Console

---

Architecture Overview

---

Modules

• carrera-common: common code for other modules, such as encapsulate zk operations.

• carrera-producer: message producer proxy with built-in Thrift Server, responsible for forwarding message from SDK client to broker.

• carrera-consumer: message consumer proxy with built-in Thrift Server, provide SDK PULL and HTTP PUSH for message consumption.

• carrera-chronos: delay message module, use RocksDB as storage engine.

• carrera-sdk: producer and consumer SDK, support Java/C/C++/Go/PHP/Python.

• rocketmq: based on RocketMQ (Ver 4.2.0)，add new features such as broker auto fail-over.

• carrera-console: a Spring-based User Web Console.

• carrera-monitor: consumer lag monitor and DDMQ cluster monitor.

• carrera-docker: provide a DDMQ docker image that runs in standalone mode.

---

Quick Start

We provide a standalone version of DDMQ as Docker image, read this for more information.

---

Deployment

Dependencies

• 64bit OS, Linux/Unix/Mac
• 64bit JDK 1.8+
• Maven 3.2.x
• MySQL 5.7.x
• Tomcat 7/8/9
• Zookeeper 3.4.x

Deployment Procedures

• Deploy MySQL & Zookeeper

  Install MySQL 5.7: https://dev.mysql.com/doc/mysql-installation-excerpt/5.7/en/

  Install Zookeeper 3.4.x: https://zookeeper.apache.org/doc/r3.1.2/zookeeperStarted.html

• Deploy RocketMQ

  Guide: RocketMQ

• Init MySQL Tables

  Guide: MySQL

• Deploy Console

  Guide: Console

• Init Zookeeper Node and Data

  call console api：

  • curl http://console_addr:8080/carrera/api/odin/internal/v4/initZkPath
  • curl http://console_addr:8080/carrera/api/odin/internal/v4/initAllZk

• Deploy Producer Proxy

  Guide: Deploy PProxy

• Deploy Consumer Proxy

  Guide: Deploy CProxy

• Deploy Chronos

  Guide: Deploy Chronos

---

Usage

• read DDMQ Console Manual for more information.

---

Contributing

Welcome to contribute by creating issues or sending pull requests. See Contributing Guide for guidelines.

---

Contact

Feel free to contact DDMQ developers via [email protected]

---

License

DDMQ is licensed under the Apache License 2.0. See the LICENSE file.

ddmq's Issues

有无qq交流群

有无qq交流群？您的微信群已满，进不去了。

DDMQ User Registration [Company Name + Website]

LAG数与MAXOFFSET不符合

LAG数与MAXOFFSET不符合，LAG数大于MAXOFFSET

DDMQ SDK 有没有C# 版本

微软在2014年已经开源.NET Core, 目前在最新一期的TechEmpower基准测试第十七轮测试 https://www.techempower.com/benchmarks/#section=data-r17&hw=ph&test=plaintext ， 17轮的测试使用的是ASP.NET Core 2.1，性能已经属于顶级的了，而且还是跨平台的。 期待能够提供C# 版本的SDK

高可用问题

proxy集群的节点高可用是怎么保障

交流群二维码过期

交流群二维码过期了，请问还有什么方式进入组织吗

延时消息有性能数据吗

[carrera-chronos] Is it supported to cancel delay msg?

如果延时消息没有过期或者失效，用户可以取消吗？如果不可以，未来有支持这个功能的想法吗？这虽然可能违背了消息队列的设计理念，但是如果支持这个功能，有些场景的业务逻辑会变得比较简单

群二维码过期了，更新一下。

群二维码过期了，更新一下。

cannot build ddmq:1.0 images

Unable to find image

Chronos 内置的时间轮服务是哪个代码？

这个os.detected.classifier找不到，怎么编译的

    <dependency>
        <groupId>io.netty</groupId>
        <artifactId>netty-tcnative</artifactId>
        <version>1.1.33.Fork22</version>
        <classifier>${os.detected.classifier}</classifier>
    </dependency>

Docker build失败

[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  1.367 s
[INFO] Finished at: 2020-09-24T12:59:23+08:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal on project carrera-console: Could not resolve dependencies for project com.xiaojukeji.ddmq:carrera-console:war:1.0-SNAPSHOT: Could not find artifact com.xiaojukeji.ddmq:carrera-common:jar:1.0.0-SNAPSHOT -> [Help 1]                                                       
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException
cp: 无法获取'carrera-console/target/carrera.war' 的文件状态(stat): 没有那个文件或目录

PProxy启动失败

2020-07-02 14:58:50,651 [ WARN ] ZkServiceImpl - no node, path=/pproxy/127.0.0.1:9613
2020-07-02 14:58:50,652 [ ERROR ] LogUtils - ERROR_TAG:ConfigManager.start, EXCEPTION:null, proxy config is illegal
2020-07-02 14:58:50,652 [ ERROR ] LogUtils - ERROR_TAG:ProxyApp.start, EXCEPTION:null, config manager start failed

go的sdk缺少文件

"go.intra.xiaojukeji.com/golang/commons/dhttp"
"go.intra.xiaojukeji.com/golang/commons/util"
log "go.intra.xiaojukeji.com/golang/commons/dlog"
"go.intra.xiaojukeji.com/foundation/didi-standard-lib/scheduler/golang"
在用go的sdk时编译时找不到这些文件

Chronos节点down机，RocksDB中的延迟消息数据怎么处理？

当一个Chronos节点消费了延迟消息并存在了RocksDB中，如果这个节点down机后，那么还未发送的延迟消息是怎么处理的？

消息吞吐量有性能数据吗

高可用问题

proxy集群的节点高可用是怎么保障

code:13, msg:FAILED: Topic is not in the proxy white list, uniqDelayMsgId:

code:13, msg:FAILED: Topic is not in the proxy white list, uniqDelayMsgId:

需要在哪里配置白名单啊？谢谢！

Does java sdk or any other sdk support sending transactional message? Thanks

docker image build error!

exec :sh build.sh
the result is like below:
[ERROR] Failed to execute goal on project carrera-common: Could not resolve dependencies for project com.xiaojukeji.ddmq:carrera-common:jar:1.0.0-SNAPSHOT: Could not find artifact com.xiaojukeji.rocketmq:rocketmq-tools:jar:4.2.2-SNAPSHOT -> [Help 1]

carrera-sdk中没有node.js 目前不支持嘛？

node需要自己去写的SDK么

Is go-sdk pure golang or dep c-lang?

Is go-sdk pure golang or dep c-lang?
请问是纯go的客户端还是依赖c？

carrera-sdk可以支持dotnet吗？

carrera-sdk可以支持dotnet吗？

There is a risk of SQL injection.

DDMQ/rocketmq/broker/src/main/java/org/apache/rocketmq/broker/transaction/jdbc/JDBCTransactionStore.java

Line 128 in 2f30b61

statement.execute(sql);

It is recommended to replace java.sql.Statement with java.sql.PreparedStatement.

配置生产代理出错

在配置完carrera-producer如果启动control.sh start
日志里输出这个
ConfigManager - checkAndUpdateKafkaConfig rmq config is null or empty, illegal
请问 rmq config在哪儿配置

编译依赖缺失