baidu / tera

An Internet-Scale Database.

Tera - An Internet-Scale Database

Copyright 2015, Baidu, Inc.

Tera is a high performance distributed NoSQL database, which is inspired by google's BigTable and designed for real-time applications. Tera can easily scale to petabytes of data across thousands of commodity servers. Besides, Tera is widely used in many Baidu products with varied demands，which range from throughput-oriented applications to latency-sensitive service, including web indexing, WebPage DB, LinkBase DB, etc. (中文)

Features

• Linear and modular scalability
• Automatic and configurable sharding
• Ranged and hashed sharding strategies
• MVCC
• Column-oriented storage and locality group support
• Strictly consistent
• Automatic failover support
• Online schema change
• Snapshot support
• Support RAMDISK/SSD/DFS tiered cache
• Block cache and Bloom Filters for real-time queries
• Multi-type table support (RAMDISK/SSD/DISK table)
• Easy to use C++/Java/Python/REST-ful API

Data model

Tera is the collection of many sparse, distributed, multidimensional tables. The table is indexed by a row key, column key, and a timestamp; each value in the table is an uninterpreted array of bytes.

• (row:string, (column family+qualifier):string, time:int64) → string

To learn more about the schema, you can refer to BigTable.

Architecture

Tera has three major components: sdk, master and tablet servers.

• SDK: a library that is linked into every application client to access Tera cluster.
• Master: master is responsible for managing tablet servers and tablets, automatic load balance and garbage collection of files in filesystem.
• Tablet Server: tablet server is the core module in tera, and it uses an enhance Leveldb as a basic storage engine. Tablet server manages a set of tablets, handles read/write/scan requests and schedule tablet split and merge online.

Building blocks

Tera is built on several pieces of open source infrastructure.

• Filesystem (required)

  Tera uses the distributed file system to store transaction log and data files. So Tera uses an abstract file system interface, called Env, to adapt to different implementations of file systems (e.g., BFS, HDFS, HDFS2, POXIS filesystem).

• Distributed lock service (required)

  Tera relies on a highly-available and persistent distributed lock service, which is used for a variety of tasks: to ensure that there is at most one active master at any time; to store meta table's location, to discover new tablet server and finalize tablet server deaths. Tera has an adapter class to adapt to different implementations of lock service (e.g., ZooKeeper, Nexus)

• High performance RPC framework (required)

  Tera is designed to handle a variety of demanding workloads, which range from throughput-oriented applications to latency-sensitive service. So Tera needs a high performance network programming framework. Now Tera heavily relies on Sofa-pbrpc to meet the performance demand.

• Cluster management system (not necessary)

  A Tera cluster in Baidu typically operates in a shared pool of machines that runs a wide variety of other distributed applications. So Tera can be deployed in a cluster management system Galaxy, which uses for scheduling jobs, managing resources on shared machines, dealing with machine failures, and monitoring machine status. Besides, Tera can also be deployed on RAW machine or in Docker container.

Documents

• Developer Doc

Quick start

• How to build

  Use sh ./build.sh to build Tera.

• How to deploy

  Pseudo Distributed Mode

  Build on Docker

• How to access

  teracli

  API

Contributing to Tera

Contributions are welcomed and greatly appreciated.

Read Roadmap to get a general knowledge about our development plan.

See Contributions for more details.

Follow us

To join us, please send resume to tera-user at baidu.com.

tera's Issues

sync API issue

• 用户自定义超时未实现
• 内部batch导致高延迟

merge的一个问题（多个tablet进行merge可能导致时序混乱）

a和b进行merge，先分别执行unload
b的unload失败进而重新被load起来成为ready状态，
接着b和c被选中进行merge，分别进行unload，
之后a的unload成功，发现b仍处于unload状态（此时b的unload是为c而准备的，但a不知道，以为b还在为a而unload，因而将自己转为OnMerge状态，苦苦等待着b的unload结束开始merge） 😢

b和c的unload成功后merge出d来，b就从tablet界消失了，a却一直处于OnMerge状态。。。好可怜的a啊，求程序猿解救！

对已超时的读请求直接丢掉，节省资源消耗

@00k 提出，客户端发起读请求没返回，重试时，从ts看，rpc队列里会存在多个相同的读请求，此时执行最后一个就可以了
一般ts read pending由于资源不足导致，所以重试会加重资源不足，引起连锁反应
@taocp

尝试使用tera过程中遇到的问题，请赐教

几个概念问题：
1、api描述中，LG，CF，都是什么概念？划分原则是什么样的？
2、mutation Reader 的读写接口中 的参数qualifier是什么？
3、scan接口描述中
在设置区间返回后，做了下面两个add
desc->AddColumnFamily("family21");
desc->AddColumn("family22", "qualifier22");
是指定scan的列么？
如果不指定，可以scan整行么？
scan多列之后，如何从result_stream中读某列数据。
请在api描述中解释一下，非常感谢。

Flash cache垃圾收集会清理正在使用的sst文件

复现方式：一个ts，一个tablet，连续写几小时，leveldb日志里就会出现copy to local

修复流式scan

• 给出当前scan单tablet的吞吐
• 预期下流式scan的吞吐
• 验证修复后的流式scan性能

2 issues about CopyToLocal

1. parallel limitation for copy-to-local, to avoid impact to dfs
2. read pending number will go high while copy-to-local, and may triger loadl balance to make the condition worser

tera.flag这个magic file一直没出现。

编译后没法启动。连-h都没法用。

race condition in some master callback funcs

example：
https://github.com/BaiduPS/tera/blob/master/src/master/master_impl.cc#L1945

if a TS load a tablet success and down, master receives the response and run the callback function but doesn't find ts is down because of message delay, and changes the status of tablet to kReady, then no ts serve the tablet

疑似bug，两个lg只写其中一个，不删log

manifest write into meta table

avoid too many open files; lower random write to dfs

系统是否也可以发布二进制版本

1、源代码安装
2、二进制安装

这个神奇的core后面肯定隐藏着一个大bug

分布式文件系统不稳定的时候&用了tcmalloc会触发

(gdb) bt
0 tcmalloc::CentralFreeList::FetchFromOneSpans (this=0xe7a8a0, N=24, start=0x59485b10, end=0x59485b18) at src/central_freelist.cc:301
1 0x0000000000b86bb9 in tcmalloc::CentralFreeList::FetchFromOneSpansSafe (this=0xe7a8a0, N=24, start=0x59485b10, end=0x59485b18) at src/central_freelist.cc:282
2 0x0000000000b86c96 in tcmalloc::CentralFreeList::RemoveRange (this=0xe7a8a0, start=0x59485b10, end=0x59485b18, N=24) at src/central_freelist.cc:264
3 0x0000000000b8e53f in tcmalloc::ThreadCache::FetchFromCentralCache (this=0xf89d20, cl=Variable "cl" is not available.
) at src/static_vars.h:60
4 0x0000000000b851b3 in (anonymous namespace)::do_malloc_no_errno (size=Variable "size" is not available.
) at src/thread_cache.h:364
5 0x0000000000b92cb5 in tc_malloc (size=8) at src/tcmalloc.cc:1119
6 0x0000003f0bfaf59a in operator new () from /usr/lib64/libstdc++.so.6
7 0x00007fbdb43d26fb in bfs::FileInfo::mutable_name (this=0x1905140) at src/proto/file.pb.h:510
8 0x00007fbdb43cf863 in bfs::FileInfo::MergePartialFromCodedStream (this=0x1905140, input=0x594862d0) at src/proto/file.pb.cc:338
9 0x00007fbdb43cab58 in google::protobuf::internal::WireFormatLite::ReadMessageNoVirtualbfs::FileInfo (input=0x594862d0, value=0x1905140)
at ../../../third-64/protobuf/include/google/protobuf/wire_format_lite_inl.h:488
10 0x00007fbdb43ab933 in bfs::ListDirectoryResponse::MergePartialFromCodedStream (this=0x5ee8eea0, input=0x594862d0) at src/proto/nameserver.pb.cc:3013
11 0x00007fbdb43df0fc in google::protobuf::MessageLite::ParseFromCodedStream (this=0x5ee8eea0, input=0x594862d0) at google/protobuf/message_lite.cc:121
12 0x00007fbdb43df3de in google::protobuf::MessageLite::ParseFromZeroCopyStream (this=0x5ee8eea0, input=Variable "input" is not available.
) at google/protobuf/message_lite.cc:170
13 0x00007fbdb447adf9 in sofa::pbrpc::RpcClientImpl::DoneCallback () from ./bin/bfs_wrapper.so
14 0x00007fbdb44a0ba3 in boost::detail::function::void_function_obj_invoker1<boost::_bi::bind_t<void, boost::_mfi::mf2<void, sofa::pbrpc::RpcClientImpl, google::protobuf::Message*, sofa::pbrpc::shared_ptrsofa::pbrpc::RpcControllerImpl const&>, boost::_bi::list3boost::_bi::value<sofa::pbrpc::shared_ptr<sofa::pbrpc::RpcClientImpl >, boost::bi::valuegoogle::protobuf::Message*, boost::arg<1> ()()> >, void, sofa::pbrpc::shared_ptrsofa::pbrpc::RpcControllerImpl const&>::invoke () from ./bin/bfs_wrapper.so
15 0x00007fbdb448e048 in sofa::pbrpc::RpcControllerImpl::Done () from ./bin/bfs_wrapper.so
16 0x00007fbdb448ffb1 in sofa::pbrpc::RpcClientStream::on_received () from ./bin/bfs_wrapper.so
17 0x00007fbdb44a8e57 in sofa::pbrpc::RpcMessageStreamsofa::pbrpc::shared_ptr<sofa::pbrpc::RpcControllerImpl >::on_read_some () from ./bin/bfs_wrapper.so
18 0x00007fbdb44a16bf in boost::asio::asio_handler_invoke<boost::asio::detail::binder2<boost::_bi::bind_t<void, boost::_mfi::mf2<void, sofa::pbrpc::RpcByteStream, boost::system::error_code const&, unsigned long>, boost::bi::list3boost::bi::value<sofa::pbrpc::shared_ptr<sofa::pbrpc::RpcByteStream >, boost::arg<1> ()(), boost::arg<2> ()()> >, boost::system::error_code, unsigned long> > ()
from ./bin/bfs_wrapper.so
19 0x00007fbdb44a17a1 in boost::asio::detail::reactive_socket_recv_op<boost::asio::mutable_buffers_1, boost::_bi::bind_t<void, boost::_mfi::mf2<void, sofa::pbrpc::RpcByteStream, boost::system::error_code const&, unsigned long>, boost::_bi::list3boost::bi::value<sofa::pbrpc::shared_ptr<sofa::pbrpc::RpcByteStream >, boost::arg<1> ()(), boost::arg<2> (*)()> > >::do_complete ()
from ./bin/bfs_wrapper.so
20 0x00007fbdb448a463 in sofa::pbrpc::ThreadGroupImpl::thread_run () from ./bin/bfs_wrapper.so
21 0x0000003f0b90610a in start_thread () from /lib64/tls/libpthread.so.0
22 0x0000003f0b0c5ee3 in clone () from /lib64/tls/libc.so.6
23 0x0000000000000000 in ?? ()

编译不过

猜测是执行build.sh ?
libsofa-pbrpc.a(buffer.o):(.data.rel.ro._ZTVN4sofa5pbrpc11WriteBufferE[ZTVN4sofa5pbrpc11WriteBufferE]+0x38)：对‘google::protobuf::io::ZeroCopyOutputStream::WriteAliasedRaw(void const, int)’未定义的引用
libsofa-pbrpc.a(gzip_stream.o):(.data.rel.ro._ZTVN4sofa5pbrpc30AbstractCompressedOutputStreamE[ZTVN4sofa5pbrpc30AbstractCompressedOutputStreamE]+0x38)：对‘google::protobuf::io::ZeroCopyOutputStream::WriteAliasedRaw(void const, int)’未定义的引用
libsofa-pbrpc.a(gzip_stream.o):(.data.rel.ro._ZTVN4sofa5pbrpc16GzipOutputStreamE[ZTVN4sofa5pbrpc16GzipOutputStreamE]+0x38)：对‘google::protobuf::io::ZeroCopyOutputStream::WriteAliasedRaw(void const, int)’未定义的引用
libsofa-pbrpc.a(block_wrappers.o):(.data.rel.ro._ZTVN4sofa5pbrpc28BlockCompressionOutputStreamE[ZTVN4sofa5pbrpc28BlockCompressionOutputStreamE]+0x38)：对‘google::protobuf::io::ZeroCopyOutputStream::WriteAliasedRaw(void const, int)’未定义的引用
libsofa-pbrpc.a(block_wrappers.o):(.data.rel.ro._ZTVN4sofa5pbrpc18SnappyOutputStreamE[ZTVN4sofa5pbrpc18SnappyOutputStreamE]+0x38)：对‘google::protobuf::io::ZeroCopyOutputStream::WriteAliasedRaw(void const, int)’未定义的引用
libsofa-pbrpc.a(block_wrappers.o):(.data.rel.ro._ZTVN4sofa5pbrpc15LZ4OutputStreamE[ZTVN4sofa5pbrpc15LZ4OutputStreamE]+0x38): more undefined references to `google::protobuf::io::ZeroCopyOutputStream::WriteAliasedRaw(void const, int)' follow
collect2: error: ld returned 1 exit status
make: *** [sofa-pbrpc-client] 错误 1
另外build.sh 写的太挫

poor performance of sequential write

due to leveldb, even strictly sequential write would cause lots of compaction

有一个神奇的core

0 0x000000000099a847 in leveldb::TableIter::Valid (this=0x7f732788d550) at table/table.cc:115
1 0x000000000099a264 in leveldb::TableIter::value (this=0x7f732788d550) at table/table.cc:125
2 0x000000000099605a in leveldb::(anonymous namespace)::IterWrapper::value (this=0x7f7170971a30) at table/merger.cc:44
3 0x0000000000995ffa in leveldb::(anonymous namespace)::MergingIterator::value (this=0x7f714c2c1cb0) at table/merger.cc:231
4 0x00000000009c7040 in leveldb::DBImpl::DoCompactionWork (this=0x7f7325f0a4d0, compact=0x7f724e7bacd0) at db/db_impl.cc:1170
5 0x00000000009c5c32 in leveldb::DBImpl::BackgroundCompaction (this=0x7f7325f0a4d0) at db/db_impl.cc:871
6 0x00000000009c54a7 in leveldb::DBImpl::BackgroundCall (this=0x7f7325f0a4d0) at db/db_impl.cc:786
7 0x00000000009c540b in leveldb::DBImpl::BGWork (db=0x7f7325f0a4d0) at db/db_impl.cc:778
8 0x00000000009b8f60 in leveldb::ThreadPool::BGThread (this=0xe9e588) at util/thread_pool.cc:163
9 0x00000000009b8825 in leveldb::ThreadPool::BGThreadWrapper (arg=0xe9e588) at util/thread_pool.cc:46
10 0x0000003f0b90610a in start_thread () from /lib64/tls/libpthread.so.0
11 0x0000003f0b0c5ee3 in clone () from /lib64/tls/libc.so.6
12 0x0000000000000000 in ?? ()

有时候也会core在112行，反正都是Valid

简易版快照回滚

支持无merge情况下快照回滚，对性能暂无要求

• 快照work
• 支持回滚

incremental GC

对于已经ready的tablet进行gc，不需要全量ready

• 将原来的全局gc从master里抽出来
• 添加新的增量gc策略作为可选项，供大家试用
• 替换掉原来的全局gc

Remove binary fle: https://github.com/BaiduPS/tera/blob/master/tera/leveldb/libsnappy.a

We cloud build that from source. this hurt portability.

Master启动时会core掉

应该是这个提交引入的
#166

通过Docker镜像部署tera

既然开源，技术文档可以多讲下设计原因。

一些地方直接就给出个结论。

多进程调用客户端接口，为什么在子进程中不能往下运行，会在打开table的时候挂起信号量，而在父进程中完全没有问题。

master先启动时ts初始化没完成导致不响应命令被kick

master先启动后，会让ts load meta表，ts如果还没init完成，没有自己的session_id，会以seesion_id不匹配的原因拒绝load，然后导致master kick ts。

客户端show出的size不准

没有计算level0的大小，导致show出来的和实际值差距非常大。
可以考虑计算level0的大小，也计算memtable的大小。
或者给出两个值：预估的数据量大小&DFS空间占用大小。
需要解决的问题

1. 什么时候开始出问题的
2. 怎么实现精确统计

flash_env issue

往flash上充sst cache失败后会一直读dfs，不会再尝试充一次

• local失败处理
• dfs失败处理

增加配置，当本地cache路径不存在或不可访问时，直接退出

某些高并发场景对读延时有高要求，这时如果发生掉盘等异常，会使读延时增大非常多，此时勉强提供服务不如直接干掉

Load过程中recover失败会delete dbtable，触发shutdown，有时候死锁，有时候core

shutdown1过程中，拿不到某个锁，DBTable或是DBImpl，导致夯住。
另一种情况core
0 0x0000000000956232 in leveldb::VersionSet::LastSequence (this=0x2bf7a420105ff) at ./db/version_set.h:214
1 0x00000000009a4c11 in leveldb::DBImpl::GetLastVerSequence (this=0x7f8faabdf600) at db/db_impl.cc:1678
2 0x0000000000942de5 in leveldb::DBTable::GarbageClean (this=0x7f8e87bcff50) at db/db_table.cc:1112
3 0x000000000093c65a in leveldb::DBTable::Shutdown1 (this=0x7f8e87bcff50) at db/db_table.cc:135
#4 0x000000000093d1dd in ~DBTable (this=0x7f8e87bcff50) at db/db_table.cc:183

5 0x0000000000939c9d in leveldb::DB::Open (options=@0x7f8ebcc38ec0, dbname=@0x7f8ebcc38e88, dbptr=0x7f8ebcc38fd8) at db/db.cc:56
6 0x00000000007f78c8 in tera::io::TabletIO::Load (this=0x7f8ebcc38e30, schema=@0x7f8fba285a30, key_start=@0x7f8f571474c0, key_en

多列场景下随机读优化

采用多次seek的方式降低cpu消耗

批量插入数据接口出core(20150820下载编译的版本）

每次出core的地方还不一样

kv模式scan少数据

代码需要增加注释。

不论是内部编码规范还是开源实现。尤其是G家的开源实现。
代码注释往往比代码都多。尤其是.h文件中。

Tera::Client析构时不会清理环境，导致客户程序coredump

client在析构时，后台线程还在运行，会使用一些已经不存在的对象。
ubuntu上编译的teracli在运行后退出时出core

Completion of error handling

Would you like to add more error handling for return values from functions like the following?

• pthread_mutex_init ⇒ Mutex constructor
• pthread_cond_wait ⇒ CondVar::Wait

链接与snappy的问题

src/leveldb/Makefile

db_bench tera_bench 容易出现/usr/bin/ld: cannot find -lsnappy的问题

不影响teracli和tera_main的生成。

SDK在tablet恢复后，写入无法恢复

tablet在一段时间不可用后，sdk的拉长重试会导致及时tablet恢复了，sdk还会等待很长时间。

build失败后不支持断点续传

build失败后再次尝试会重新下载所有的依赖包。

tera是不是也可以构建版本了。

不然不知道新提交的代码做了什么fix或升级。

master操作日志持久化

load/unload/split/merge等过程需要持久化，否则master重启之后可能丢失状态，导致某些tablet不被服务

Master在人工leave safemode时会core

Remove unnecessary null pointer checks

An extra null pointer check is not needed in functions like the following.

• HashClient
• MasterImpl
• MemTableOnLevelDB
• TableImpl

多线程，跑异步接口，大多数情况下会崩溃

如题。请问这是什么原因？

Upgrade split

• 将原分裂流程的ts写meta表，更改到master改meta表；
• ts端完成卸载旧表，加载两新子表（为后续高负载时，降低分裂时延做铺垫）
• 通过master端分裂前记日志，保证master宕机重启，仍能完成分裂的重试；
• 定期收集分裂key剥离

TERA 不就是HBase的C 版本吗！

TERA 不就是HBase的C++ 版本吗！

客户端sdk可以不包括任何编译依赖

比如用户使用pb3，但是sdk和tera内部使用pb2.
这就影响用户使用了。
这块速度解决下把。

write performance issue caused by pread in compaction

assume a pread cost 0.1ms for reading 1k data, the maximum read throughput will be 10MB/s

BUILD说明可以更专业化一些

1、依赖哪些外部基础库。
2、这些基础库的下载地址直接给出
3、依赖库是否有版本要求。比如pb是否可以使用pb3.
4、完整的安装过程示例以及结果提示。
如configure --help 。安装到哪儿、成功是什么样子的。
5、系统如何启动。
6、系统如何控制。

文档有一点不准确的地方

https://github.com/BaiduPS/tera/blob/master/doc/a_brief_intro_of_tera_(in_chinese).pdf

1. 序号有点乱，不信看目录
2. 4.1.1表格管理中，信息有误，tera目前没有使用root -> meta -> user tablets三层结构，而是meta -> user tablets.

readme描述可能导致的误解

https://github.com/BaiduPS/tera/blob/master/README.md#L32 (系统依赖 “使用分布式文件系统（HDFS、NFS等）持久化数据与元信息” 这一句）

此处的NFS是指baidu私有的nfs，
公众看到”NFS“可能首先会联想到同样是分布式文件系统的NFS.
会不会造成歧义？如果是，建议把“NFS"这个词删掉。

另外，感觉是不是还需要说明下，代码里虽然有适配nfs文件系统的地方，但那个nfs是百度自研的系统，不是sun的那个。
免得用户鼓捣半天以为tera可以构建在sun的那个NFS上。

本周要准备的文档

1. 构建指导
2. 搭建手册
3. 配置说明
4. 配置实例