easonyi / paxosstore

PaxosStore has been deployed in WeChat production for more than two years, providing storage services for the core businesses of WeChat backend. Now PaxosStore is running on thousands of machines, and is able to afford billions of peak TPS.

PaxosStore

PaxosStore is a distributed-database initially inspired by Google MegaStore. It's the second generation of storage system developed to support current WeChat sevice and applications. PaxosStore has been deployed in WeChat production for more than two years, providing storage services for the core businesses of WeChat backend including user account management, user relationship management (i.e., contacts), instant messaging, social networking (i.e., Moments), and online payment (i.e., WeChat Pay).

Now PaxosStore is running on thousands of machines, and is able to afford billions of peak TPS.

Prior to PaxosStore, we have been using a QuorumKV storage system to support various WeChat services with strongly consistent read/write since 2011. As the number of storage servers rises to tens of thousand, the operational mantenance and development of a NWR-based system in such a large scale become painful. That's why we come up with PaxosStore: a new generation of distributed database, built on top of the leaseless Paxos consensus layer, providing

• Two paxos consensue libraries (the key algorithms described in our paper published at the VLDB 2017 are now open source):

  • Certain for the general PaxosLog + DB design;
  • PaxosKV optimized for key-value storage (PaxosLog-as-value);

  In addition, the following items are planned for open source by October 2017.

• A high performance key-value system

• A system that supports rich data structures such as queues, list, set and collections

• A high performance storage engine backed by LSM-tree

• A New SQL-like Table system

The PaxosStore Architecture

Please refer to the following publications for the technical details of PaxosStore.

• VLDB 2017 PaxosStore: High-availability Storage Made Practical in WeChat
• 微信PaxosStore：深入浅出Paxos算法协议
• 微信PaxosStore内存篇：十亿Paxos/分钟的挑战
• 微信后台基于时间序的海量数据冷热分级架构设计实践

Build

• Certain
• PaxosKV

License

PaxosStore is under the BSD 3-Clause License. See the LICENSE.txt file for details.