Kafka Topics, Partitions and Offsets

In previous article, we learned the basic Kafka flow:

Producer
   |
   v
 Kafka
   |
   v
Consumer

We also introduced three very important concepts:

Topic
Partition
Offset

These concepts are so fundamental that almost everything else in Kafka builds on top of them.

If you understand these three properly, concepts such as:

• consumer groups,
• ordering,
• message keys,
• scalability,
• consumer lag,
• replay,
• replication,
• and parallel processing

become much easier to understand.

So in this chapter, we are going to slow down and look carefully at how Kafka actually organizes records.

1. Start with the Big Picture

Suppose we have an e-commerce system.

Whenever a customer places an order, Order Service publishes an event:

OrderCreated

The high-level flow looks like this:

But saying:

“The event is stored in Kafka”