3. Kafka Fundamentals: Let’s Build It From Scratch
In the last blog, we explored different approaches to help our services work independently.
We started with a database, explored message queues using BullMQ, discovered fan-out with RabbitMQ, and finally arrived at event streaming.
We ended with a question: How does Apache Kafka actually make event streaming possible?
Imagine our ShopNow application has finally started using Kafka.
When a customer places an order, our order service publishes an OrderCreated event to Kafka.
The payment service, inventory service, and analytics service can then consume that event independently.
Sounds simple enough. But let’s pause for a moment.
What actually happens when our order service sends an event to Kafka?
What is an event? Where does that event go? Where is it stored? How do other services find it? And how does Kafka handle millions of events without everything becoming a bottleneck?
Instead of memorizing Kafka’s architecture, let’s figure out what we would need to build such a system ourselves.
1. First, What Are We Actually Sending?
Let’s return to our ShopNow application.
Whenever something happens in our application, we may want to share that information with other services.
For example, when a customer places an order, our application could produce an event like this:
{
"event": "OrderCreated",
"orderId": "ORD-101",
"userId": "USER-42",
"amount": 1499,
"currency": "INR"
}This is just a JSON object describing something that happened. Our application could produce similar events for other situations.
When a payment succeeds:
{
"event": "payment-success",
"paymentId": "PAY-501",
"orderId": "ORD-101",
"amount": 1499,
"status": "SUCCESS"
}
When inventory is reserved:
{
"event": "inventory-reserved",
"orderId": "ORD-101",
"productId": "PROD-25",
"quantity": 2
}
When a customer registers:
{
"event": "user-signup",
"userId": "USER-42",
"email": "customer@example.com",
"registeredAt": "2026-10-25T10:30:00Z"
}Notice something interesting.
All these records are JSON objects, but they describe different things that happened in our application.
Now imagine ShopNow grows and starts generating thousands or even millions of these records.
Where should we put them?
Should we throw every event into one giant collection, mixing orders, payments, inventory changes, and customer registrations together?
We could, but finding and organizing related events would become difficult for new consumer who joins later.
We need a way to group related events into separate streams. This brings us to our next idea.
2. Second, How Should We Organize These Events?
Imagine arranging all our events into groups based on what they represent.
- Order-related events go into an
ordersstream. - Payment-related events go into a
paymentsstream. - Inventory-related events go into an
inventorystream. - Customer-related events go into a
customersstream.
In Kafka, these named streams are called topics.
For example, our orders topic might contain records like these:
{
"event": "OrderCreated",
"orderId": "ORD-101",
"amount": 1499
},
{
"event": "OrderShipped",
"orderId": "ORD-103",
"trackingId": "TRK-9001"
},
{
"event": "OrderCancelled",
"orderId": "ORD-102",
"reason": "Customer request"
}These events describe different things that happened to orders, so we might organize them in the same topic.
A topic is a named stream of records. It doesn’t require every record to have exactly the same JSON structure, although designing consistent event schemas is important.
We’ve now solved our first problem: we have a way to organize related events.
But another problem appears as our application grows.
What happens when our orders topic starts receiving millions of records?
Can we distribute the storage and workload across multiple machines instead of relying on one long log handled by a single machine?
Let’s figure that out next.
3. Third, What Happens When Millions of Events Arrive?
Till now, we’ve organised events from different categories into their respective logical divisions called topics.
Now, let’s continue designing Kafka by focusing on a single orders topic.
Remember, every time an order is placed, ShopNow publishes an event to the orders topic.
Now, imagine ShopNow has gone viral across the globe. We’re receiving millions of order events from different countries.
All these events are going into the same orders topic.
Suppose you open the admin dashboard and want to answer questions like:
- How many orders came from India today?
- Which country generated the highest revenue?
- How many orders were placed in the US in the last hour?
To answer these questions, our analytics consumer needs to process order events and calculate the required statistics.
But here’s the problem: if all these events are kept in one enormous sequence, processing this growing stream can become a challenge.
As the volume of events grows, we need a better way to organise and process the stream.
What if we could divide the events into multiple smaller sequences based on our requirements?
Let’s understand this with an example. An order event might look something like this:
{
"event": "OrderCreated",
"orderId": "ORD-101",
"country": "IN",
"amount": 1499
}
Another order might come from the US:
{
"event": "OrderCreated",
"orderId": "ORD-102",
"country": "US",
"amount": 2500
}Notice that each event contains a country code.
What if we could organise these events based on the country they belong to?
For example, we could use the country code as the key when publishing each event.
Conceptually, our orders topic could be divided like this:
orders topic
|
┌───────────┼───────────┐
| | |
v v v
Partition 0 Partition 1 Partition 2
Key: IN Key: US Key: UK
India US UK
India US UK
India US UKThis is a simplified illustration. In a real Kafka setup, a partition can contain events from multiple countries.
Now, our analytics consumer can process events from these partitions and calculate country-wise order statistics.
But here’s the interesting part.
We decide how to divide our event stream based on our requirements.
In our example, we used the country code as the key because we wanted to organise events by country. In another system, we might use a user ID, an order ID, or some other key, depending on what we’re trying to achieve.
We also decide how many partitions our topic should have.
Kafka gives us the building blocks, but choosing the number of partitions and deciding how events should be distributed across them are design decisions we make when building our system.
These smaller, independently ordered sequences within a topic are called partitions.
But Which Partition Should Receive an Event? — hashed key approach.
We’ve divided our orders topic into three partitions.
When a new event arrives, should it go to Partition 0, Partition 1, or Partition 2?
We need a rule to make that decision.
One simple approach is to distribute events in a round-robin fashion: the first event goes to Partition 0, the next to Partition 1, then Partition 2, and the cycle repeats.
But there’s another possibility.
What if we want events belonging to the same order to go to the same partition? This would be useful when we want to preserve the order of events for that partition.
Kafka supports this through a concept called a partition key.
Let’s first understand something important: a partition key isn’t a special field that must be added to your event. It’s simply a piece of information that the producer chooses to use when deciding which partition should receive the event.
Look at our event again:
{
"event": "OrderCreated",
"orderId": "ORD-101",
"userId": "user-109"
"country": "IN",
"amount": 1499
}The orderId, country, and amount are all pieces of information in the event.
If our goal is to group events by country, we could choose country as the partitioning key.
If our goal is to keep events belonging to the same order together, we could choose orderId instead.
The information itself isn’t special. It’s called a partition key because we’re using it to determine which partition receives the event.
For example, a producer could publish an event using ORD-101 as its key:
Key: ORD-101
Value: OrderCreated eventThe producer can hash the key and use the result to determine which partition receives the event.
Think of the process like this:
Event
|
v
Message Key
ORD-101
|
v
Hash Key
|
v
Select Partition
|
v
Append EventWhen multiple events use the same key, they are generally directed to the same partition, provided the partitioning configuration remains unchanged.
This is useful because Kafka preserves the order of records within a partition, but not across different partitions.
And if a producer doesn’t provide a key, Kafka can use a different partitioning strategy.
Notice how these two ideas work together:
- Topics organise related events into named streams.
- Partitions divide a topic into multiple ordered sequences.
- Partition keys help determine which partition receives an event.
We’ve now figured out how a single topic can be divided and how events can be assigned to its partitions.
4. We Need A Machine
So far, we’ve discovered two ways to organise our data:
- Topics group related events.
- Partitions divide a topic into multiple ordered sequences of records.
But we’ve only discussed how to organise our data. We haven’t answered a fundamental question yet.
Where do these events actually live?
A topic is a logical concept. A partition defines an ordered sequence of event logically distributed across multiple partitions. But neither of these is a physical machine.
Our events still need disk space to be stored. And when a producer sends an event or a consumer requests records, some machine needs to handle that request.
Let’s think about how we would design this.
First, We Need a Machine to Store Our Data
Imagine we’re building Kafka from scratch, and we have just one machine available.
We install Kafka on this machine and let it store our partition data and handle requests from producers and consumers.
This machine can store partitions belonging to multiple topics. For example, it could store partitions from both our orders and users topics.
One Kafka Server
Broker 1
┌─────────────────────┐
│ │
│ Topic: orders │
│ ┌───────────────┐ │
│ │ Partition 0 │ │
│ ├───────────────┤ │
│ │ Partition 1 │ │
│ └───────────────┘ │
│ │
│ Topic: users │
│ ┌───────────────┐ │
│ │ Partition 0 │ │
│ ├───────────────┤ │
│ │ Partition 1 │ │
│ └───────────────┘ │
│ │
└─────────────────────┘A server running Kafka that stores and serves partition data of multiple topic is called a broker.
Notice that a broker doesn’t belong to just one topic. It can store multiple partitions from multiple topics.
Let’s Play With Our Single Broker
Before we move forward, let’s understand how our design works with just one broker.
Imagine our Kafka server is running, and we’ve created two topics: orders and users. For now, let's focus only on the orders topic, which has three partitions.
Whenever a new order is placed, the producer sends an event to Kafka. Kafka’s producer-side partitioning logic determines which partition receives that event.
Let’s publish a few events and see what happens.
Event 1: A customer places an order.
{
"event": "OrderCreated",
"orderId": "ORD-101",
"amount": 1499
}Suppose this event is assigned to Partition 0.
Event 2: Another customer places an order.
{
"event": "OrderCreated",
"orderId": "ORD-102",
"amount": 2500
}This time, suppose the event goes to Partition 1.
Event 3: A third order arrives.
{
"event": "OrderCreated",
"orderId": "ORD-103",
"amount": 999
}Suppose it goes to Partition 0 again.
Our single broker now stores the following partition data:
Broker 1
┌──────────────────────┐
│ │
│ orders topic │
│ │
│ Partition 0 │
│ ┌────────────────┐ │
│ │ ORD-101 │ │
│ │ ORD-103 │ │
│ └────────────────┘ │
│ │
│ Partition 1 │
│ ┌────────────────┐ │
│ │ ORD-102 │ │
│ └────────────────┘ │
│ │
│ Partition 2 │
│ ┌────────────────┐ │
│ │ (empty) │ │
│ └────────────────┘ │
│ │
└──────────────────────┘The partition assignments here are just an example; the actual destination depends on the producer’s partitioning strategy.
Notice something important: each partition maintains its own ordered sequence of records.
When a new event is assigned to Partition 0, Kafka appends it to the end of that partition’s sequence. It doesn’t insert the event between older records.
This is why we describe a Kafka partition as an append-only log.
Think of it like a journal. You keep adding new entries at the end, one after another. The previous entries remain in their original positions while they’re retained.
Each record also has an offset, which identifies its position within that partition.
For example:
Partition 0
Offset 0 → ORD-101
Offset 1 → ORD-103Offsets are local to each partition. Partition 1 has its own offsets, starting from 0 for this simplified example.
Get Vikash Gupta’s stories in your inbox
Join Medium for free to get updates from this writer.
So, what have we learned from our little experiment?
- A producer publishes an event to a topic.
- The producer’s partitioning logic determines which partition receives it.
- The broker stores the partition’s records as an append-only sequence.
- Each record has an offset within its partition.
That’s the basic picture of Kafka working with a single broker.
But now let’s ask ourselves one last question before we move forward.
What happens when one broker is no longer enough — or when the machine itself fails?
That’s the next problem we need to solve as we continue designing Kafka.
5. What If Our Single Broker Goes Down?
Let’s recap what we’ve built so far.
We have one Kafka broker storing our orders topic, which contains multiple partitions. Whenever a producer publishes an event, it gets assigned to a partition and is appended to the end of that partition's sequence.
Everything works fine so far.
But now, let’s think about what happens when our system grows.
Our single broker is handling all the incoming events, storing all the records, and serving requests from consumers. As traffic and data grow, we might want to distribute this workload across multiple machines.
We can add more brokers and distribute our partitions among them.
But there’s another problem we need to consider.
What happens if the machine storing our data suddenly crashes?
Imagine our broker goes offline because of a hardware failure or some other issue.
Our producers can no longer publish events to that broker, and consumers may no longer be able to read the partitions it was serving.
Even though our data is divided into multiple partitions, all those partitions still depend on the same machine.
We’ve organised our data, but we haven’t protected it from machine failure.
So, how could we solve this? Think about it from first principles.
If losing one machine makes our data unavailable, what if we keep another copy of the same partition on a different machine?
That way, if one machine goes down, another machine may still have the data we need.
- This idea of keeping copies of partition data on different machines is called replication.
- And to make this possible, we need multiple Kafka brokers working together as one system. Such a group of brokers is called a Kafka cluster.
Let’s explore how Kafka distributes partitions across brokers and keeps replicas available when a broker fails.
6. How Does Kafka Keep Another Copy of Our Data?
In the previous section, we discovered two important ideas:
- Kafka cluster: Multiple brokers working together as one system.
- Replication: Keeping copies of the same partition on different brokers to help handle machine failures.
Now, let’s understand how Kafka makes replication work.
Two Copies, One Source of Truth
Imagine we have two brokers, and both store a copy of Partition 0.
Broker 1 Broker 2
┌──────────────┐ ┌──────────────┐
│ │ │ │
│ Partition 0 │ │ Partition 0 │
│ │ │ │
│ ORD-101 │ │ ORD-101 │
│ ORD-103 │ │ ORD-103 │
│ │ │ │
└──────────────┘ └──────────────┘
Copy 1 Copy 2Now, suppose a new order event arrives.
Should the producer write it to both brokers independently?
That could create problems. What if one broker receives the event but the other doesn’t? What if they receive events in a different order?
We need a clear rule for coordinating these copies.
Kafka solves this by assigning different roles to the replicas of a partition.
One replica becomes the leader, while the other replicas act as followers.
The Leader and the Followers
Let’s say Broker 1 holds the leader replica of Partition 0, while Broker 2 holds a follower replica.
Broker 1 Broker 2
┌──────────────┐ ┌──────────────┐
│ │ │ │
│ Partition 0 │ │ Partition 0 │
│ LEADER │ ────────> │ FOLLOWER │
│ │ Replicate│ │
└──────────────┘ records └──────────────┘
▲
│
New events
│
ProducerThe leader handles writes for that partition. Followers fetch records from the leader to keep their copies up to date.
Kafka also has a term for replicas that are keeping up with the leader: In-Sync Replicas (ISR). These are replicas that are sufficiently caught up with the leader to be considered in sync.
So, when a producer publishes a new event to Partition 0, it is written through the leader replica. The followers fetch and replicate that record, and those that remain sufficiently caught up can be part of the ISR.
This gives Kafka a coordinated way to maintain multiple copies of the same partition and helps it decide which replicas are eligible to take over if the leader fails.
What If the Leader Broker Fails?
Now let’s return to our original problem. Suppose Broker 1 goes offline.
The leader replica of Partition 0 is no longer available. But Broker 2 still has a follower replica.
If Kafka has a suitable in-sync replica available, it can elect a new leader from the eligible replicas. Producers and consumers can then continue using the partition through its new leader.
Broker 1 Broker 2
OFFLINE NEW LEADER
┌──────────────┐ ┌──────────────┐
│ │ │ │
│ Partition 0 │ │ Partition 0 │
│ LEADER │ │ LEADER │
│ │ │ │
└──────────────┘ └──────────────┘
▲
│
Requests continueThis is a simplified illustration. In a real cluster, whether a replica can become leader depends on Kafka’s replication state and configuration.
The important idea is that a broker failure doesn’t necessarily make a partition unavailable because another broker may have a usable copy.
That’s the benefit of replication.
7. Let’s Connect the Pieces — Checkpoint 1
We’ve now discovered how the pieces of Kafka’s architecture fit together:
- Topics organise related events.
- Partitions divide a topic into ordered sequences of records.
- Brokers store and serve partition data.
- Clusters allow multiple brokers to work together as one system.
- Replication keeps copies of partitions on different brokers.
- Leaders and followers coordinate how those copies are written and maintained.
We’ve built a basic picture of how Kafka stores and protects event data.
But we still haven’t answered another important question from the beginning of our journey:
How do our services actually read these events, keep track of their progress, and resume reading if they restart?
Let’s explore how consumers work in Kafka next.
8. How Do Consumers Read Events and Remember Their Progress?
First, Let’s Read Some Events
Imagine our ShopNow application has an analytics service that wants to read events from the orders topic.
Our topic contains these records:
Partition 0
Offset 0 → ORD-101
Offset 1 → ORD-103
Offset 2 → ORD-105
Offset 3 → ORD-107
Offset 4 → ORD-109
...The analytics service connects to Kafka and starts reading records from the partition.
Suppose it has processed records up to offset 2.
So far, everything looks fine.
But now, imagine the analytics service suddenly crashes.
When it restarts, how will it know where to continue?
Should it start reading from offset 0 and process everything again? Or should it continue from where it stopped?
We need a way to remember its progress.
The Problem: Who Remembers the Last Read Offset?
One simple idea would be to let the analytics service save its last processed offset in its own memory.
But what happens when the service crashes? Its in-memory state disappears.
We could save the offset in a database, but then every service would need to manage its own progress-tracking mechanism.
And what if we run multiple instances of the same service to process events in parallel? How do we coordinate their progress and decide which instance should read which partition?
We need a better approach.
What if Kafka could remember the reading progress for a group of consumers, rather than making every consumer manage everything independently?
This brings us to the idea of a consumer group.
Meet the Consumer Group
In Kafka, consumers can work together as part of a consumer group.
Think of a consumer group as a team of consumers working together to read a topic.
For example, our analytics service might have a consumer group called analytics-group.
Kafka Cluster
|
orders topic
|
┌────────┴────────┐
| |
Partition 0 Partition 1
| |
v v
Consumer 1 Consumer 2
| |
└────────┬────────┘
|
analytics-groupHere, two consumers from the same group can read different partitions in parallel.
Kafka coordinates the assignment of partitions to consumers within the group. This allows us to add more consumers and distribute the work.
But there’s an important rule:
Within one consumer group, a partition is assigned to at most one consumer at a time.
For example, Consumer 1 can read Partition 0 while Consumer 2 reads Partition 1. They don’t both get assigned Partition 0 at the same time.
What if we have three partitions but only two consumers? One consumer can be assigned multiple partitions.
And what if we add more consumers than there are partitions? Some consumers will have no partition assigned until the assignments change.
This is how Kafka distributes partition-reading work within a consumer group.
How Does Kafka Remember Our Progress?
Now let’s return to our original problem.
Suppose analytics-group is reading Partition 0.
The consumer processes records at offsets 0, 1, and 2. It then commits its progress to Kafka.
Conceptually, Kafka maintains a mapping like this:
Consumer Group Topic Partition Committed Offset
----------------------------------------------------------------
analytics-group orders Partition 0 3
analytics-group orders Partition 1 7Notice something important.
Kafka doesn’t maintain just one offset for the entire consumer group. It tracks progress separately for each topic partition.
In this simplified example, the committed offset 3 means the group can resume reading Partition 0 from offset 3 next.
Why 3 and not 2?
Because Kafka consumers commonly commit the next offset to read, rather than the offset of the last record processed.
Now, suppose Consumer 1 crashes.
When a consumer in the same group takes over Partition 0, it can retrieve the group’s committed offset and resume from the appropriate position.
The consumer doesn’t have to start from the beginning, provided the required records are still available and the progress was committed.
One important detail: Kafka’s committed offset represents the progress the consumer has saved, not necessarily every record it has already processed. If a consumer crashes before committing its latest progress, some records may be processed again.
That’s why consumers should be designed to handle retries and possible duplicate processing.
Why Do We Need Consumer Groups?
Consumer groups solve two related problems.
1. Remembering progress
Kafka stores committed offsets for each consumer group’s topic partitions, so consumers can resume from saved progress after restarts or reassignment.
2. Processing in parallel
Multiple consumers in the same group can read different partitions at the same time, allowing work to be distributed across consumer instances.
There’s one more useful consequence.
Imagine both our analytics service and inventory service need to read the same orders topic.
They can use different consumer groups:
orders topic
|
┌─────────┴─────────┐
| |
analytics-group inventory-group
| |
Analytics Service Inventory ServiceEach group maintains its own reading progress. The analytics service can read an event without moving the inventory service’s committed offset.
So, different consumer groups can independently read the same events, while consumers within one group share the work across partitions.
9. Let’s Connect the Pieces — Checkpoint 2
We’ve now added another important part to our Kafka architecture:
- Topics organise related events.
- Partitions divide topics into ordered sequences.
- Brokers store and serve partition data.
- Replication keeps copies of partition data on different brokers.
- Consumer groups coordinate consumers that work together to read partitions.
- Committed offsets let each consumer group remember its progress for each partition.
We’ve now seen both sides of Kafka’s basic journey: producers write events, and consumers read them while keeping track of their progress.
Thank you for reading! See you in the next part!
