Kafka from scratch (YaK)
A from-scratch distributed publish-subscribe queue in the spirit of Apache Kafka: three replicated brokers, a ZooKeeper-style coordinator with leader election, and producer/consumer clients, all over HTTP.
The brief
YaK (“Yet another Kafka”) recreates the core of a Kafka-style log. Producers publish messages to named topics, messages are stored across three replicated brokers, and consumers subscribe and read. A ZooKeeper-style service keeps the cluster healthy: it polls the brokers on an interval, maintains the live-broker list, and elects a leader, so the system tolerates a broker failure without losing the topic.
Architecture
- Brokers. Each broker is a Flask app on its own port with its own on-disk store and an append-only logfile recording every action with timestamp, topic and ID. Three brokers replicate for fault tolerance.
- The coordinator. A ZooKeeper-style Flask service polls broker health, tracks liveness, and performs leader election. Clients ask it where to read and write, which decouples them from broker topology.
- Clients. Producer and consumer CLIs with retry-on-failure HTTP loops, so a transient broker death shows up as a retry, not a crash.
Why it holds up
The interesting property is what happens when you kill a broker mid-stream:
the poller notices, the leader re-elects, and the clients’ retry loops carry
on. Building that behaviour from plain Flask and requests, with no framework
doing the thinking, is exactly the kind of systems exercise the course wanted.
Credit
Team build; repo owned by teammate Manas Chebrolu.