isabella232 / leadership

Distributed Leader Election using docker/libkv

Leadership: Distributed Leader Election for Clustered Environments.

Leadership is a library for a cluster leader election on top of a distributed Key/Value store.

It is built using the docker/libkv library and is designed to work across multiple storage backends.

You can use leadership with Consul, etcd and Zookeeper.

// Create a store using pkg/store.
client, err := store.NewStore("consul", []string{"127.0.0.1:8500"}, &store.Config{})
if err != nil {
	panic(err)
}

underwood := leadership.NewCandidate(client, "service/swarm/leader", "underwood", 15*time.Second)
electedCh, _ := underwood.RunForElection()

for isElected := range electedCh {
	// This loop will run every time there is a change in our leadership
	// status.

	if isElected {
		// We won the election - we are now the leader.
		// Let's do leader stuff, for example, sleep for a while.
		log.Printf("I won the election! I'm now the leader")
		time.Sleep(10 * time.Second)

		// Tired of being a leader? You can resign anytime.
		candidate.Resign()
	} else {
		// We lost the election but are still running for leadership.
		// `elected == false` is the default state and is the first event
		// we'll receive from the channel. After a successful election,
		// this event can get triggered if someone else steals the
		// leadership or if we resign.

		log.Printf("Lost the election, let's try another time")
	}
}

It is possible to follow an election in real-time and get notified whenever there is a change in leadership:

follower := leadership.NewFollower(client, "service/swarm/leader")
leaderCh, _ := follower.FollowElection()
for leader := range leaderCh {
	// Leader is a string containing the value passed to `NewCandidate`.
	log.Printf("%s is now the leader", leader)
}
log.Fatal("Cannot follow the election, store is probably down")
// Recovery code or exit

A typical use case for this is to be able to always send requests to the current leader.

Fault tolerance

Leadership returns an error channel for Candidates and Followers that you can use to be resilient to failures. For example, if the watch on the leader key fails because the store becomes unavailable, you can retry the process later.

func participate() {
    // Create a store using pkg/store.
    client, err := store.NewStore("consul", []string{"127.0.0.1:8500"}, &store.Config{})
    if err != nil {
        panic(err)
    }

    waitTime := 10 * time.Second
    underwood := leadership.NewCandidate(client, "service/swarm/leader", "underwood", 15*time.Second)

    go func() {
        for {
            run(underwood)
            time.Sleep(waitTime)
            // retry
        }
    }()
}

func run(candidate *leadership.Candidate) {
    electedCh, errCh := candidate.RunForElection()
    for {
        select {
        case isElected := <-electedCh:
            if isElected {
                // Do something
            } else {
                // Do something else
            }

        case err := <-errCh:
            log.Error(err)
            return
        }
    }
}

License

leadership is licensed under the Apache License, Version 2.0. See LICENSE for the full license text.