a3y3 / pegasusdb

Distributed and fault tolerant database

Hi there 👋

My name is Soham Dongargaonkar and I'm a software engineer at Visa.

• 🔭 I’m currently working on:

  • Working at Suki AI
  • Creating a Logger that allows you to indent logs at different levels, just the way you indent code
  • Creating a Kafka like distributed pub/sub system from scratch in Rust

• 🌱 Check out some of my blog posts:

  • Building my own distributed Key/Value Store
  • Save time by deploying your microservices to GCP in parallel
  • How to add Readiness Checks for Google App Engine on Django

• 📫 Find me on:

  • My website
  • LinkedIn
  • Twitter

pegasusdb's Issues

Fix a bug that can deadlock a newly elected leader

When the old leader is partitioned off (or crashes), the newly elected leader can have values surfaced on applyCh - but this can cause a problem since no RPC is waiting to consume them (the old leader's RPC already did that).
Thus, every leader also needs to verify if someone is even expecting that value before going into a wait.

Use condition variables to provide strong linearizability

• For each leader RPC (get/putappend), wait on cond.
• When a value appears on applyCh, if server is leader (or maybe even if it isn't?), set kv.appliedIndex and kv.appliedId values, then broadcast.
• When a thread awakens from wait, check if kv.appliedIndex is the same as the value returned from Start(). If it's the same, check kv.appliedid. If it's the same, all went well. Else if it's less, wait. else, something went wrong, and the command wasn't applied (and will never be applied), so we must retry.
• Broadcast only when at least one thread is waiting (producer/consumer problem)

When a leader accepts a log entry, make sure it is replicated

Identify each client request individually

To handle dropped RPCs, the first step is to identify each req with a unique id.

Condense code for Get and PutAppend()

the RPC handlers Get and PutAppend() pretty much do the exact same thing, and so the functionality can be condensed to a single method.

When a duplicate request is received, ignore it

Once each request is identified uniquely, the kv server needs to reject a request that it has seen already.
This can be done by a simple hashmap if we also track each clerk by a client_id:

• for each clerk, store the last id it sent.
• once a clerk sends a new id, we can safely delete the old id (since this implies the client saw the response for the previous request)
• if the clerk sends the same request twice, we want to respond on THIS thread. So we can simply skip the insert, and go straight to waiting for this id to be posted on this index.
• Nope - since the producer thread only sends a message ONE time, the old thread is the one that gets that message. Hence, we should store the result in the Request thread itself, so if the duplicate request is seen again, we just return the stored result directly.

Fix log entries from previous terms never being committed

If a leader replicates a non committed log entry from a previous term, it can't commit it until there is a newer log entry in the current term. However, a client that's waiting on that entry to be committed is never going to submit a new command (to ensure linearizability).

This can be fixed if a thread in the Clerk continuously sends an idempotent command (aka GET) to the leader. Doing this will ensure that there is always at least one entry in the current term in the current leader, which will force an entry (if any) from the previous term to be committed and sent back to the clerk.

Parallelize loop in updateLeader()

The loop in updateLeader() needs to be parallelized so that it's faster (and also because if it's not, the main thread could get hung up calling a crashed/partitioned leader)

[Clerk] Send operation to current leader

• Get a list of all kvservers
• Contact each server to see who's the leader
• If no one is a leader, retry after a pause
• If more than one leader, log.fatalf

Once a message appears on applyCh, apply it to the current state machine.

• Listen on applyCh
• Create a hashmap from string to string
• For each message, use the contents to update the internal hashmap

When an old leader detects a new one, ask the Clerk to resend the req to the new leader

When partitions heal, the old leader will get a HB from the new leader and revert to the follower state. However, the old leader may still have RPC threads waiting for an entry to come on applyCh.
An easy solution for this would be to have the threads check if the server they're running on is still the leader once they awaken. If they find out they aren't, they can easily return an error to the clerk. However, for this, they should awaken in the first place - hence, all follower KV servers should still consumer.cond.Broadcast() in case they were a leader at some time and had RPC threads waiting.