pikalabs / floyd Goto Github PK
View Code? Open in Web Editor NEWA raft consensus implementation that is simply and understandable
License: GNU General Public License v3.0
A raft consensus implementation that is simply and understandable
License: GNU General Public License v3.0
multi group can break through the big lock in floyd. However, we need consider how to set different key in multi group?
For simplicity, we haven't add no-op entry when the server become leader. As the raft's paper said, it will cause inconsistency problem.
we have supported batch to optimize the performance, yet pipeline is also needed.
centos 7.2
在顺利编译完floyd、pink、slash和rocksDB后,编译example中simple和redis的时候报错:
Makefile:9: Warning: slash path missing, using default
Makefile:16: Warning: slash path missing, using default
Makefile:23: Warning: rocksdb path missing, using default
g++ -pg -g -O2 -ggdb3 -pipe -fPIC -W -Wwrite-strings -Wpointer-arith -Wreorder -Wswitch -Wsign-promo -Wredundant-decls -Wformat -D_GNU_SOURCE -D__STDC_FORMAT_MACROS -std=c++11 -gdwarf-2 -Wno-redundant-decls -Wno-unused-variable -DROCKSDB_PLATFORM_POSIX -DROCKSDB_LIB_IO_POSIX -DOS_LINUX -I../../.. -I/root/downloads/floyd/floyd/example/simple/third/slash -I/root/downloads/floyd/floyd/example/simple/third/pink -I/root/downloads/floyd/floyd/example/simple/third/rocksdb/include -o t t.o -I../../../ -I../../third/rocksdb/include -I../../third/slash/ -I../../third/pink/ -L../../lib/ -L../../third/slash/slash/lib/ -L../../third/rocksdb/ -L../../third/pink/pink/lib/ -lfloyd -lpink -lslash -lrocksdb -lsnappy -lprotobuf -lz -lbz2 -lrt -lssl -lcrypto -lpthread
../../third/rocksdb//librocksdb.a(db_impl.o):在函数‘ZSTD_Supported’中:
/root/downloads/floyd/floyd/third/rocksdb/./util/compression.h:84:对‘ZSTD_versionNumber’未定义的引用
../../third/rocksdb//librocksdb.a(column_family.o):在函数‘ZSTD_Supported’中:
/root/downloads/floyd/floyd/third/rocksdb/./util/compression.h:84:对‘ZSTD_versionNumber’未定义的引用
/root/downloads/floyd/floyd/third/rocksdb/./util/compression.h:84:对‘ZSTD_versionNumber’未定义的引用
../../third/rocksdb//librocksdb.a(format.o):在函数‘ZSTD_Uncompress’中:
/root/downloads/floyd/floyd/third/rocksdb/./util/compression.h:775:对‘ZSTD_createDCtx’未定义的引用
/root/downloads/floyd/floyd/third/rocksdb/./util/compression.h:778:对‘ZSTD_decompress_usingDict’未定义的引用
/root/downloads/floyd/floyd/third/rocksdb/./util/compression.h:779:对‘ZSTD_freeDCtx’未定义的引用
../../third/rocksdb//librocksdb.a(format.o):在函数‘LZ4_Uncompress’中:
/root/downloads/floyd/floyd/third/rocksdb/./util/compression.h:611:对‘LZ4_createStreamDecode’未定义的引用
/root/downloads/floyd/floyd/third/rocksdb/./util/compression.h:618:对‘LZ4_decompress_safe_continue’未定义的引用
/root/downloads/floyd/floyd/third/rocksdb/./util/compression.h:619:对‘LZ4_freeStreamDecode’未定义的引用
/root/downloads/floyd/floyd/third/rocksdb/./util/compression.h:611:对‘LZ4_createStreamDecode’未定义的引用
/root/downloads/floyd/floyd/third/rocksdb/./util/compression.h:618:对‘LZ4_decompress_safe_continue’未定义的引用
/root/downloads/floyd/floyd/third/rocksdb/./util/compression.h:619:对‘LZ4_freeStreamDecode’未定义的引用
/root/downloads/floyd/floyd/third/rocksdb/./util/compression.h:614:对‘LZ4_setStreamDecode’未定义的引用
/root/downloads/floyd/floyd/third/rocksdb/./util/compression.h:614:对‘LZ4_setStreamDecode’未定义的引用
../../third/rocksdb//librocksdb.a(block_based_table_builder.o):在函数‘ZSTD_Compress’中:
/root/downloads/floyd/floyd/third/rocksdb/./util/compression.h:738:对‘ZSTD_compressBound’未定义的引用
/root/downloads/floyd/floyd/third/rocksdb/./util/compression.h:742:对‘ZSTD_createCCtx’未定义的引用
/root/downloads/floyd/floyd/third/rocksdb/./util/compression.h:745:对‘ZSTD_compress_usingDict’未定义的引用
/root/downloads/floyd/floyd/third/rocksdb/./util/compression.h:746:对‘ZSTD_freeCCtx’未定义的引用
../../third/rocksdb//librocksdb.a(block_based_table_builder.o):在函数‘LZ4_Compress’中:
/root/downloads/floyd/floyd/third/rocksdb/./util/compression.h:546:对‘LZ4_compressBound’未定义的引用
/root/downloads/floyd/floyd/third/rocksdb/./util/compression.h:551:对‘LZ4_createStream’未定义的引用
/root/downloads/floyd/floyd/third/rocksdb/./util/compression.h:559:对‘LZ4_compress_fast_continue’未定义的引用
/root/downloads/floyd/floyd/third/rocksdb/./util/compression.h:565:对‘LZ4_freeStream’未定义的引用
../../third/rocksdb//librocksdb.a(block_based_table_builder.o):在函数‘LZ4HC_Compress’中:
/root/downloads/floyd/floyd/third/rocksdb/./util/compression.h:665:对‘LZ4_compressBound’未定义的引用
/root/downloads/floyd/floyd/third/rocksdb/./util/compression.h:670:对‘LZ4_createStreamHC’未定义的引用
/root/downloads/floyd/floyd/third/rocksdb/./util/compression.h:671:对‘LZ4_resetStreamHC’未定义的引用
/root/downloads/floyd/floyd/third/rocksdb/./util/compression.h:676:对‘LZ4_loadDictHC’未定义的引用
/root/downloads/floyd/floyd/third/rocksdb/./util/compression.h:681:对‘LZ4_compress_HC_continue’未定义的引用
/root/downloads/floyd/floyd/third/rocksdb/./util/compression.h:687:对‘LZ4_freeStreamHC’未定义的引用
../../third/rocksdb//librocksdb.a(block_based_table_builder.o):在函数‘LZ4_Compress’中:
/root/downloads/floyd/floyd/third/rocksdb/./util/compression.h:554:对‘LZ4_loadDict’未定义的引用
collect2: 错误:ld 返回 1
make: *** [t] 错误 1
Maybe the commit index shouldn't be persistent in RaftMeta, instead, it will be set correctly by new leader after the first log entry being appended on quorum.
libprotoc 3.4.0 版本编译提示错误
请问需要的版本为多少呢
以上问题已经解决,
执行了下./proto/pr.sh
提一个建议,觉得应该在makefile之前执行下 ./proto/pr.sh 不然容易产生版本不一致的情况
Use rocksdb::WriteBatch to remove all useless log in one batch, it better than calling rocksdb::Delete one by one
src/http_conn.cc: 在成员函数‘bool pink::HttpConn::BuildRequestHeader()’中:
src/http_conn.cc:333:32: 错误:从类型‘int64_t* {aka long long int*}’到类型‘long int*’中的 static_cast 无效
static_cast<long*>(&tmp));
support thread safe write like leveldb.
Down or slow nodes will cause huge drops in performance, which becomes more and more serious as time goes by.
The PeerThread for abnormal nodes need more and more time to get entries which growth over time.
Meanwhile, the global context mutex hold by it and prevent any other PeerThreads from advancing commit index.
So the performance goes worse and worse at the perspective of user, and finally not available.
In our use case, sometime get/set interface can't not satisfy our requirements. Such as in pika_hub, efan. We want using floyd choosing leader, so it has best to have a lock distribute lock interface.
leader will print logs of error logs when there is a node down. We need optimize the log there.
Maybe we can refer to the implementation of Raft by etcd.
site: https://github.com/coreos/etcd/tree/master/contrib/raftexample
A new server who is not added into cluster yet, will constantly send RequestVote RPC with new term number to others, and this will cause new raft election.
But the new server couldn't receive any AppendEntry RPC, so it will timeout and redo the same process, then cause a new election process again and again.
Certainly, this will result in very poor availability.
right now the log in floyd is logger, it is origin copy from leveldb's log, there is some weakness in it, such us when we restart the node, the old log will be delete, don's support split log by date, and so on..
since we don't pay too much attention on it as we develop in the early phase, I think we need optimize right now
在并行请求RequestVoteRPC的时候,目前的实现是只要有一个peer返回VOTE_DENIED, 就从candidate变为follower, 但是根据论文里面的实现,不是只要多数返回了VOTE_GRANTED就可以变为LEADER么? 这里是不是返回VOTE_DENIED的时候简单记录下就好?谢谢
every node has peer threads even it becomes follower, it is better to quit the peer threads when the node become follower and start the peer threads when it become leader again. Since the only leader requires peer threads to do heartbeat.
right now there is not enough case to test floyd's implementation of raft is safety, we should add some more extreme test case, or we should add some test tool like jepsen?
right now the read operation need Append Entry RPC, in raft's paper and paxos paper, both of them suggest leader lease to optimize the read operation. we can call it raft with lease.
Diego's phd paper has suggest a simple way to support membership change, every time we can only add/remove a single node in to cluster, in this limitation, this problem become simple.
there is some variable name that's not the same as raft's paper, such as last_apply_index. in raft paper it is last_applied.
By doing that, we can make the code more readable
Lines 555 to 557 in 01a6979
we want test some extreme case such as 5 node have 5 different logs, then they will choose the one who has the largest term or if they have the same term, then the one who have longest index as the leader.
Right now, we test these case by run simple test case, and maybe there will exist the case that nodes have different logs, but this is not good enough. We need a more accurate test case
从api列表上看只是支持kv形式的读写,有开放的接口让用户自己定义在raft状态机成功之后的行为吗?如果有很多复杂的逻辑不能通过读写kv来执行。
kv_server.cc:156:31: error: ‘class floyd::Floyd’ has no member named ‘DirtyWrite’
Status result = floyd_->DirtyWrite(request.key(), request.value());
kv_server.cc:139:47: error: no matching function for call to ‘floyd::Floyd::GetServerStatus(std::string&)’
bool ret = floyd_->GetServerStatus(value);
lots of bug exists
v2.0.5 is not compatible with v2.0.3 in proto as follow:
message Entry {
enum OpType {
kRead = 0;
kWrite = 1;
kDelete = 2;
kTryLock= 4;
kUnLock = 5;
}
required OpType optype = 1;
// used in key value operator
required uint64 term = 2;
required string key = 3;
optional bytes value = 4;
// used in lock and unlock
optional bytes holder = 5;
optional uint64 lease_end = 6;
}
Line 146 in d3af033
注释和论文里面有写:
// if votedfor is null or candidateId, and candidated's log is at least as up-to-date
// as receiver's log, grant vote
但是这个rpc实现逻辑里面,好像完全没有去判断FloydContext内部的vote_for_ip和vote_for_port, 事实是如果一个follower变成candidate后,首先会把vote_for_ip和vote_for_port设置成自己;此时如果收到RequestVoteRPC请求过来,应该直接return false, 表明自己这一轮已经投过票了(投给自己)
不知道我理解的对不对?谢谢
I have wrote a simple test to get the performance of floyd
write 10000 cost time microsecond(us) 8136502, qps 1229
Node | Role | Term | CommitIdx | Leader | VoteFor | LastLogTerm | LastLogIdx | LastApplyIdx |
127.0.0.1:8901 leader 2 10000 127.0.0.1:8901 127.0.0.1:8901 2 10000 10000
127.0.0.1:8902 follower 2 10000 127.0.0.1:8901 127.0.0.1:8901 2 10000 10000
127.0.0.1:8903 follower 2 10000 127.0.0.1:8901 127.0.0.1:8901 2 10000 10000
127.0.0.1:8904 follower 2 10000 127.0.0.1:8901 127.0.0.1:8901 2 10000 10000
127.0.0.1:8905 follower 2 10000 127.0.0.1:8901 127.0.0.1:8901 2 10000 10000
write 10000 cost time microsecond(us) 8325906, qps 1201
Node | Role | Term | CommitIdx | Leader | VoteFor | LastLogTerm | LastLogIdx | LastApplyIdx |
127.0.0.1:8901 leader 2 20000 127.0.0.1:8901 127.0.0.1:8901 2 20000 20000
127.0.0.1:8902 follower 2 20000 127.0.0.1:8901 127.0.0.1:8901 2 20000 20000
127.0.0.1:8903 follower 2 20000 127.0.0.1:8901 127.0.0.1:8901 2 20000 20000
127.0.0.1:8904 follower 2 20000 127.0.0.1:8901 127.0.0.1:8901 2 20000 20000
127.0.0.1:8905 follower 2 20000 127.0.0.1:8901 127.0.0.1:8901 2 20000 20000
it's qps is about 1000+, it is to low, so we had better to improve it
Hi, 在看 Floyd 代码时发现,Floyd 将 read 当做一条日志 Append 到 peers 解决脏读的问题,在 read log entry apply 之后,(1)通过 context_->apply_cond.SignalAll();
唤醒阻塞等待在 read 的 client 的,(2)然后 client 再 read rocksdb 中 k-v 返回给用户,但是感觉这样会有一 corner case ?
上面标注的两步(1)(2)两步之间会存在一个时间空隙,尽管很小,但是在这段时候,如果针对同一个 key的一个新的 update command 被 apply,虽然没有 read 旧数据,但是 read 会读到更新的数据,
当然单 client 的情况下不会出现这种问题
Add network partition case
there is many string(ip):int(port) in code, that is ugly.
we should replace them with a struct node I think
Lines 385 to 391 in 01a6979
Since it's hard to prove that a consensus algorithm is correct, and there is joke that it harder to prove that a consensus algorithm is correct than implement a consensus algorithm.
Right now many consensus algorithm implementation will passing jepsen test to prove that it is correct, so floyd need pass jepsen test, too.
Just read floyd code and raft paper. According to raft paper, read operation will be forwarded to leader and leader holds the whole committed log, that is, the complete meta info.
So is it possible to separate read operation and write operation apart? In this case, for read operation, leader just return to the client without waiting for majority of servers' append entry response. @CatKang @baotiao
floyd/floyd/src/floyd_peer_thread.h
Line 101 in d3af033
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.