A deep dive into consensus algorithms and fundamentals of distributed computing

diagram of three people each saying “I propose,” with the words “consensus failed” superimposed in red
diagram of three people each saying “I propose,” with the words “consensus failed” superimposed in red
Image source: Analytics India Magazine (used with permission)

What are consensus algorithms?
What are distributed databases?
How do Kubernetes and ZooKeeper store data in a fault-tolerant way?

These are some of the questions that we will try to answer in this article. Part A of the article will try to explain some of the common terminologies that you will hear in the current world of distributed computing. In Part B, we will explore the basic concepts of consensus algorithms. Part C will cover two famous consensus algorithms, Raft and Zab, and towards the end, we will explore how these algorithms power the distributed nature of ZooKeeper and Kubernetes.


A bit brief about how it started —

The challenge presented before us was to migrate the core ordering application from Ruby to Java (Spring Boot). To begin with this huge task, we picked our main booking show aggregate api, which was hitting around 30 downstream api’s and serving at average throughput of 1280rpm with average latency of 1100ms

The tasks we had in-front of us were -

  1. Build a system that is fault tolerant and isolated from downstream failures
  2. Build a fault tolerant and refreshable caching mechanism

Technologies we shortlisted for this task (Prerequisites for this article)
1. Redis (https://redis.io/)
2. Netflix Hystrix (v2.0.2) (https://www.baeldung.com/spring-cloud-netflix-hystrix)
3. …

Pratyush Bansal

Inquisitive Mind. Software Engineer @Microsoft

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store