Kafka Basics Explained

As a DevOps engineer, I look at applications a bit differently than pure application developers do. I’m less focused on business logic and far more interested in how components communicate, how data flows under load, and what happens to the system when traffic suddenly spikes tenfold.

When I started understand Kafka, I wanted to understand it from that exact system-level perspective: Where does Kafka fit in the architecture, what bottleneck does it solve, and how does data move through it?

The best way to understand architectural tools is to look at the exact problem they were created to solve. So, let’s start with a real-world scenario.

Why do we actually need Kafka?

Imagine Uber has around 1 lakh cars running across different cities.

Every second, these cars are continuously sending data such as:

• Location
• Speed
• Distance travelled
• Average speed
• Pricing information
• Driver status
• Trip information etc..

Now imagine all this data is directly going into a database.

That’s where the problem starts.

Let’s say we have three different services that need this data.

For example:

• Fare calculation service
• Driver analytics service
• Customer service etc..

If every car is sending data and all three services are directly interacting with the database, the number of database operations becomes huge.

For a smaller example, imagine we have 1,000 drivers and three services.

That could mean roughly:

1,000 drivers × 3 services = 3,000 operations

And this is happening continuously.

As the number of users and vehicles increases, the database has to handle more and more operations.

The database can store a lot of data, but continuously pushing a huge amount of real-time data directly into it can become a bottleneck.

So we need something in between.

And this is where Kafka comes into the picture.

So, what does Kafka do?

The simple way I understand Kafka is:

Kafka sits between the application producing the data and the services consuming the data.

Instead of:

Application → Database

we have:

Application → Kafka → Services → Database

Kafka is designed for handling high-throughput data streams.

The database, on the other hand, is mainly responsible for storing and querying data.

So you can roughly think about it like this:

Kafka
High throughput
Event streaming
Short/controlled retention

        ↓
Database
Lower write throughput compared to Kafka
Long-term data storage
Queries and persistence

Now let’s take our Uber example again.

Uber’s application is generating all this data:

Location
Speed
Distance
Pricing
Driver status
Trip information

So Uber is the Producer.

Why?

Because it is producing the data.

The producer sends this raw data to Kafka.

For example:

Uber Application
       |
       | Raw data
       ↓
     Kafka

Kafka now has the raw events coming from the application.

The data doesn’t necessarily need to be processed before entering Kafka.

Kafka can act as the middle layer where these events are stored and made available to different services.

Consumers come into the picture

Now we have Kafka containing all this raw data.

What happens next?

Different services can consume this data from Kafka.

For example:

Kafka
                    |
        +-----------+-----------+
        |           |           |
        ↓           ↓           ↓
   Fare Service  Analytics   Customer
                 Service     Service

These are our Consumers.

Each consumer takes the data from Kafka and performs its own operation.

For example, the fare service may calculate the price.

The analytics service may calculate driver-related statistics.

Another service may use the same data for customer-related operations.

So the same raw data coming into Kafka can be consumed by different services depending on what they need.

What happens to the database now?

Instead of every single event immediately hitting the database, the consumers can process the data and then perform operations such as batch/bulk inserts depending on the application’s design.

Get Anish Bhat K’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

For example:

1 lakh events
      ↓
    Kafka
      ↓
  Consumers
      ↓
 Process data
      ↓
 Batch/Bulk insert
      ↓
   Database

So instead of constantly making individual database operations for every incoming event, we can process the events and write them to the database in batches.

This is one of the places where Kafka becomes very useful.

Now let’s understand Kafka Internals

Kafka Topics :

Once we understand Producer → Kafka → Consumer, the next question is:

Where does Kafka actually keep these events?

This is where Topics come in.

A Kafka topic is basically a category or stream where we organize related events.

Let’s take another example.

Suppose we are using a food delivery application like Swiggy or Zomato.

When I order food, there can be different types of events happening.

For example:

Order placed
Food preparation started
Food prepared
Food packed

These can belong to one topic related to the food order.

Then we can have another topic related to delivery:

Rider assigned
Rider location updated
Rider reached restaurant
Rider picked up food
Rider reached customer

So we can have something like:

Kafka
               |
       +-------+-------+
       |               |
       ↓               ↓
  Order Topic     Delivery Topic

The idea is simply to organize related events into different topics.

Simple Representation of Kafka

Partitions Inside a Kafka Topic

Now let’s come to partitions.

A Kafka topic can have multiple partitions.

There is no fixed rule for how we should partition the data. It completely depends on the project and business requirements.

For example, we can partition the data based on:

• Area
• Time zone
• Region
• City etc..

So, partitioning is something that we decide based on how we want to distribute the data.

Partitions are identified using an index:

Partition 0
Partition 1
Partition 2
Partition 3

Now comes an important point about consumers.

One consumer can consume multiple partitions, but one partition can be assigned to only one consumer within a consumer group.

For example:

Partition 0 → Consumer 1
Partition 1 → Consumer 1
Partition 2 → Consumer 2
Partition 3 → Consumer 2

Here, Consumer 1 is consuming two partitions and Consumer 2 is consuming two partitions.

This brings us to consumer groups.

Consumer Groups

A consumer group is a group of consumers that work together to consume data from Kafka.

Let’s say we have 4 partitions.

We can have 4 consumers:

Partition 0 → Consumer 1
Partition 1 → Consumer 2
Partition 2 → Consumer 3
Partition 3 → Consumer 4

But if we add a fifth consumer:

Partition 0 → Consumer 1
Partition 1 → Consumer 2
Partition 2 → Consumer 3
Partition 3 → Consumer 4
                           Consumer 5 → No partition

The fifth consumer will remain idle because there are only four partitions.

Kafka also handles the distribution of partitions among consumers. This is called rebalancing.

For example, if we initially have two consumers:

Consumer 1 → Partition 0, Partition 1
Consumer 2 → Partition 2, Partition 3

If another consumer joins, Kafka can redistribute the partitions among the consumers.

This is what we generally refer to as Kafka’s self-balancing or rebalancing.

Kafka as a Queue and Pub/Sub

Another interesting thing about Kafka is that it can be used in a queue-like way as well as a publish-subscribe model.

In a queue-like model, consumers in the same consumer group share the work.

Producer
   |
 Kafka
   |
+--+--+
|     |
C1    C2

In the publish-subscribe model, multiple consumer groups can consume the same data independently.

Our Uber example is a good example of this.

Kafka
                      |
          +-----------+-----------+
          |           |           |
          ↓           ↓           ↓
       Pricing     Analytics    Customer
       Service      Service      Service

The same data produced by Uber can be consumed by different services for different purposes.

Conclusion

So, if I have to keep Kafka very simple, I would look at it like this:

Producer → Kafka → Topic → Partitions → Consumers

The producer generates the data.

Kafka handles the incoming stream.

Topics organize the data, and partitions help distribute it.

Consumers consume and process the data based on their requirements.

And that’s the basic Kafka flow from a system and DevOps perspective.

There are many more Kafka concepts to explore, but understanding this basic flow gives us a good starting point.