A deep dive into consensus algorithms and fundamentals of distributed computing
What are consensus algorithms?
What are distributed databases?
How do Kubernetes and ZooKeeper store data in a fault-tolerant way?
These are some of the questions that we will try to answer in this article. Part A of the article will try to explain some of the common terminologies that you will hear in the current world of distributed computing. In Part B, we will explore the basic concepts of consensus algorithms. Part C will cover two famous consensus algorithms, Raft and Zab, and towards the end, we will explore how these algorithms power the distributed nature of ZooKeeper and Kubernetes.
A bit brief about how it started —
The challenge presented before us was to migrate the core ordering application from Ruby to Java (Spring Boot). To begin with this huge task, we picked our main booking show aggregate api, which was hitting around 30 downstream api’s and serving at average throughput of 1280rpm with average latency of 1100ms
The tasks we had in-front of us were -
- Build a system that is fault tolerant and isolated from downstream failures
- Build a fault tolerant and refreshable caching mechanism
Technologies we shortlisted for this task (Prerequisites for this article)
1. Redis (https://redis.io/)
2. Netflix Hystrix (v2.0.2) (https://www.baeldung.com/spring-cloud-netflix-hystrix)
3. …
Pratyush Bansal
Inquisitive Mind. Software Engineer @Microsoft
