Comments (13)
What do you mean by multiraft? We have a new raft implementation here which might be of interest: etcd-io/etcd#874
from cockroach.
Multiraft is the as-yet-unfinished raft implementation in https://github.com/cockroachdb/cockroach/tree/master/multiraft. It differs from most existing raft implementations in that it optimizes for the case where each server is a member of many (partially overlapping) consensus groups by e.g. consolidating heartbeats per pair of nodes.
from cockroach.
@bdarnell It would be great if we could work on a shared raft implementation with this feature in mind.
/cc @bmizerany @xiangli @unihorn
from cockroach.
Agreed. Please let me know what we can do to work together, if possible.
from cockroach.
@bdarnell Why you want to have some overlapped raft clusters rather than well-partitioned ones?
Consolidating heartbeats can only happens if one machine is the leader of more than two raft clusters and they have a significant number of overlapped nodes, right?
from cockroach.
@xiangli, "overlapped" in this context means that each node in the cluster
is likely participating in many raft consensus groups with any other given
node.
With cockroach, we cache rpc connections between nodes and use the periodic
heartbeat to compute clock skew and link latency. Individual raft consensus
groups communicate over these established links as necessary, but they
don't originate the heartbeats themselves. Followers in consensus groups
examine the health of connections (i.e. whether or not a heartbeat was
received within the last heartbeat interval) after their timeouts expire to
determine whether to become candidates.
On Wed, Jul 23, 2014 at 11:45 AM, Xiang Li [email protected] wrote:
@bdarnell https://github.com/bdarnell Why you want to have some
overlapped raft clusters rather than well-partitioned ones?
Consolidating heartbeats can only happens if one machine is the leader of
more than two raft clusters and they have a significant number of
overlapped nodes, right?—
Reply to this email directly or view it on GitHub
#20 (comment)
.
from cockroach.
@spencerkimball It seems like what exactly we did for our new raft. We do not trigger heartbeat/election inside raft. We do not have network layer inside raft. You can send virtual
heartbeat from any source as you like.
from cockroach.
Thanks for the interest; I'll definitely take a closer look at etcd's raft implementation (hashicorp's is on my list too). It would be great to collaborate on a single high-quality implementation.
from cockroach.
@bdarnell An update on this. We have merged our raft implementation and the separate WAL that is used by etcd over here: http://godoc.org/github.com/coreos/etcd/raft and here http://godoc.org/github.com/coreos/etcd/wal
It would be great to get some feedback and perhaps work together.
from cockroach.
@philips Thanks for letting me know the etcd raft implementation have been merged. I've looked over it and I like a lot of the abstractions here (especially the way you batch up different kinds of updates in a single Ready struct. It took me while to convince myself that that was safe, but I think it's simpler than separating update channels for different types of events). That said, there are a few big things we need that aren't there yet:
- Online membership change. It looks like the only way to change the membership of the cluster is to stop and restart all the nodes; we need the ability to add and remove nodes on the fly for rebalancing, etc.
- Storing raftLog state on disk. We can't afford to keep full snapshots in memory, and we probably can't even afford all log entries since the last snapshot. We'd like to be able to drop entries from the in-memory log as soon as they are applied, and have some sort of interface for raft to reach back out to the application to retrieve older log entries and snapshots as needed to catch up out-of-date peers.
- Coalesced heartbeats. We need to keep track of our last successful communication with each peer and only send heartbeats to those nodes we haven't heard from in a while (and we need to be able to update this per-peer timestamp from the outside so a response in one raft cluster can serve as a heartbeat for all other clusters involving that pair of nodes).
This adds up to a sizable amount of complexity, although at least some of it would be useful for etcd and other projects as well.
from cockroach.
@bdarnell We are absolutely looking at online membership change and plan on doing that in the next week or so. The other two are things we would like to do but haven't yet. Working together on either of those two would be great.
from cockroach.
- We have the code ready, just need to think through the interface
- storage/log interface is doable (you control how to get/truncate): right now it is just a wrapped slice
- that is doable. and we have the plan.
from cockroach.
I'm closing this now that we've incorporated etcd's raft implementation; I'll open new issues for tracking the remaining work.
from cockroach.
Related Issues (20)
- ccl/multiregionccl: TestMrSystemDatabase failed
- cli: TestTenantZip failed HOT 1
- kv/kvserver/reports: TestCriticalLocalitiesReportIntegration failed HOT 1
- pkg/ccl/logictestccl/tests/5node/5node_test_/5node_test_shard_4/5: pkg failed
- pkg/sql/logictest/tests/fakedist-disk/fakedist-disk_test: TestLogic_alter_primary_key failed
- roachtest: follower-reads/mixed-version/survival=region/locality=global/reads=strong failed HOT 1
- kvserver: gossip_alerts sometimes uses zero TTL HOT 2
- pkg/sql/schemachanger/schemachanger_test: TestConcurrentSchemaChanges failed HOT 7
- kv/kvserver: TestClosedTimestampFrozenAfterSubsumption failed HOT 1
- cdc query: protect all required tables under pts HOT 2
- sql: label generic query plans in statement fingerprints > explain plans pages
- Add a unit test for LDR cross-cluster writes
- sql: TestMemoryLimit failed
- crosscluster/logical: add option to LDR stmt to prevent replicating TTL deletes HOT 1
- crosscluster/logical: support node to load balancer setup HOT 3
- Sentry: txn_coord_sender.go:1013: transaction unexpectedly committed, ba: Put [/Table/127/1/×/×,/Min), EndTxn(parallel commit) [/Table/127/2/×/×], [txn: 7ff059ad]. txn: "sql txn" meta={id=7ff059ad...
- roachtest: run nightlies with assertions enabled
- sql: support syntax for SECURITY DEFINER
- sql: TestExplainRedact failed HOT 2
- ALTER TABLE ADD COLUMN add NOT NULL and UNIQUE constraints columns to DSC HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from cockroach.