Microservice leader election using etcd

According to the docs, etcd is a strongly consistent, distributed key-value store that provides a reliable way to store data that needs to be accessed by a distributed system or cluster of machines. Etcd stores data in the form of key-value pairs under prefixes.

Quite mouthful right? What matters to us is that etcd is a database whose contents are (ideally) stored in multiple places (nodes) and an etcd cluster can be used as a central place for process coordination in a distributed system. Kubernetes uses etcd to carry out its vital processes! So we know etcd is no joke. And you too can use etcd in your distributed system to perform leader elections among replicas of your microservices.

In this blog, we are going to use an etcd client written in golang to write a go program that can easily become a part of your microservice to perform leader election.

This is how our program looks, I will shortly explain what it does.

package main

import (
    "context"
    "fmt"
    "log"

    client "go.etcd.io/etcd/client/v3"
    "go.etcd.io/etcd/client/v3/concurrency"
)

func main() {

    // initialize the client
    etcdClient, err := client.New(client.Config{Endpoints: []string{"localhost:2379"}})
    if err != nil {
      log.Fatal(err)
    }
    defer etcdClient.Close() // cleanup

    // create a new session for leader election
    electionSession, err := concurrency.NewSession(etcdClient, concurrency.WithTTL(1))
    if err != nil {
      log.Fatal(err)
    }
    defer electionSession.Close() // cleanup

    election := concurrency.NewElection(electionSession, "/election-prefix")
    ctx := context.Background()

    fmt.Println("Attempting to become a leader")
    // start leader election
    if err := election.Campaign(ctx, "value"); err != nil {
      log.Fatal(err)
    }
    fmt.Println("Became a leader!")

    for {
        // forever loop prevents our program from ending
    }
}

Install etcd. And run it locally by running the command etcd in a terminal.

If we run this program in two separate terminals using go run main.go, we can see one instance becomes a leader and the other one waits for the leader to be dead/crash/vanish. As soon as I interrupt the leader process by pressing Ctrl+C the other instance becomes the leader.

Now let's study the code.

First, we initialise the etcd client using the default endpoints for the etcd cluster running on our local system.

Then we create an election session using concurrency.NewSession(). This creates a lease object (called a session lease) in the etcd cluster. The default time to live for the lease is 60 seconds. Meaning, the lease will expire if it is not renewed within 60 seconds. The time to live value can be tuned by passing concurrency.WithTTL() as a parameter to concurrency.NewSession().

In our example program, we are using concurrency.WithTTL(1) because we want the other instances to try to acquire the lease as soon as the leader dies. i.e when nobody owns the lease.

Next, calling concurrency.NewElection() creates an election struct that we will further use to start leader election. concurrency.NewElection() takes a *Session and prefix as an input. The given prefix will be used during the election process to create keys in the etcd database. Finally, calling election.Campaign() kicks off the leader election process. It essentially creates a key under the given prefix

In the leader election process, the participants attempt to create keys with the session lease. Every key has a revision number and newer keys have a higher revision number. Once a process successfully creates a key with the session lease, it becomes the leader. Other processes that, by default, have a key with a higher revision number, wait for the pre-existing keys created with the session lease to be deleted. Once the key is deleted it can proceed to create a key with the session lease and become the leader.

To stop leader election you can call election.Resign()

Shameless plug: You can use Rasputin, an open-source leader election client that sits on top of the etcd client and provides a succinct API for leader election and related chores. Using Rasputin you are relieved of the load of writing code to simple tasks like performing periodic leadership shed, checking current leadership status, and listening for leadership status changes.