Kafka Consumer Groups Explained With a Real-World Example

If you’ve worked with Kafka for a while, you’ve probably heard the term Consumer Group dozens of times.

You know that consumers belong to groups.

You know Kafka assigns partitions to consumers.

And you may even know that consumers in the same group don’t normally read the same partition at the same time.

But there’s a simpler question:

Why do consumer groups exist in the first place?

And more importantly:

What problem do they actually solve in a real system?

Let’s understand it without starting with Kafka terminology.

Imagine an Online Food Delivery System

Suppose you run a food delivery platform.

Every time someone places an order, your system creates an event:

Order #101 → Pizza → Bangalore
Order #102 → Biryani → Pune
Order #103 → Burger → Mumbai
Order #104 → Pasta → Bangalore

You publish these events to a Kafka topic called:

orders

Now your company has several systems that need these orders.

For example:

• Payment Service
• Restaurant Service
• Delivery Service
• Analytics Service

At first, this sounds simple.

But there is an important requirement:

Each system needs to process every order.

The Payment Service needs the order.

The Restaurant Service needs the order.

The Delivery Service needs the order.

And Analytics needs it too.

This is where Consumer Groups become extremely useful.

What Is a Consumer Group?

A consumer group is simply a collection of Kafka consumers that work together as a single logical subscriber.

For example:

Kafka Topic: orders

                ┌── Payment Consumer
                │
                ├── Payment Consumer
                │
                └── Payment Consumer
             Consumer Group: payments

These three consumers are part of the same group.

Kafka treats them as one logical consumer.

Now imagine another group:

 Restaurant Consumer
                │
                └── Restaurant Consumer


          Consumer Group: restaurant

The important part is this:

Different consumer groups receive the same Kafka messages independently.

So the same order can be processed by:

orders
   │
   ├── payments
   ├── restaurant
   ├── delivery
   └── analytics

This is one of the most important concepts to understand about Kafka.

Now Let’s Talk About Partitions

Kafka topics are divided into partitions.

Suppose our orders topic has 6 partitions:

orders

Partition 0
Partition 1
Partition 2
Partition 3
Partition 4
Partition 5

Why?

Because partitions allow Kafka to process messages in parallel.

Now suppose our Payment Service has 3 consumers:

Payment Group

Consumer 1
Consumer 2
Consumer 3

Kafka can distribute the partitions like this:

Consumer 1 → P0, P1
Consumer 2 → P2, P3
Consumer 3 → P4, P5

So instead of one consumer processing everything:

P0 → Consumer
P1 → Consumer
P2 → Consumer
P3 → Consumer
P4 → Consumer
P5 → Consumer

we process them in parallel.

That is the real power of consumer groups.

One Partition Is Assigned to Only One Consumer in a Group

This rule is worth remembering:

Within a consumer group, a partition is assigned to only one consumer at a time.

For example:

Topic: orders

P0 ───────→ Consumer A
P1 ───────→ Consumer A
P2 ───────→ Consumer B
P3 ───────→ Consumer B
P4 ───────→ Consumer C
P5 ───────→ Consumer C

Consumer A and Consumer B won’t both process P0 as members of the same group.

This prevents duplicate processing within the normal consumer-group model.

But there is an interesting consequence.

What Happens If I Add More Consumers?

Suppose we have:

6 partitions
3 consumers

We can roughly get:

C1 → P0, P1
C2 → P2, P3
C3 → P4, P5

Now let’s add another consumer.

6 partitions
4 consumers

Kafka may rebalance them:

C1 → P0, P1
C2 → P2, P3
C3 → P4
C4 → P5

Now each consumer has less work.

Sounds great.

So why not just keep adding consumers?

Because of another important rule:

You cannot have more active consumers doing useful partition processing than there are partitions.

Suppose we have:

6 partitions
10 consumers

Only 6 consumers can have partitions:

C1 → P0
C2 → P1
C3 → P2
C4 → P3
C5 → P4
C6 → P5

C7 → nothing
C8 → nothing
C9 → nothing
C10 → nothing

Those extra consumers are basically waiting.

This is why partition count matters when designing Kafka-based systems.

The Real-World Example Gets More Interesting

Let’s go back to our food delivery platform.

Suppose we have:

orders topic
6 partitions

And our architecture looks like this:

                         Kafka
                           │
                        orders
                           │
          ┌────────────────┼────────────────┐
          │                │                │
          ▼                ▼                ▼
       Payments         Restaurant       Analytics
        Group             Group            Group

Each group has its own independent position in the topic.

For example:

Payments Group
P0 → offset 1050
P1 → offset 1100

Restaurant Group
P0 → offset 980
P1 → offset 1020

Analytics Group
P0 → offset 700
P1 → offset 750

This is extremely powerful.

Analytics might be slower than Payments.

That’s okay.

Kafka doesn’t require every consumer group to process messages at the same speed.

Each group keeps track of its own progress.

Consumer Group Offsets

This brings us to another important concept:

Offsets.

Every message in a Kafka partition has an offset.

For example:

Partition 0

Offset     Message

100        Order #101
101        Order #102
102        Order #103
103        Order #104
104        Order #105

A consumer reads these messages sequentially.

After processing them, the consumer group keeps track of its progress.

For example:

payments group
P0 → offset 105

This essentially tells Kafka:

“The payments group has progressed this far in partition 0.”

Now imagine the consumer crashes.

When it comes back, it can continue from its committed offset instead of starting from the beginning.

What Happens When a Consumer Dies?

This is where consumer groups become really useful in production.

Suppose we have:

P0 → Consumer A
P1 → Consumer A

P2 → Consumer B
P3 → Consumer B

P4 → Consumer C
P5 → Consumer C

Suddenly:

Consumer B <dies>

What happens to P2 and P3?

Kafka detects that Consumer B is no longer part of the group.

It triggers a rebalance.

The remaining consumers receive the partitions again.

For example:

Consumer A → P0, P1, P2
Consumer C → P3, P4, P5

The exact assignment depends on the partition-assignment strategy and group state, but the important idea is:

Kafka redistributes the work.

This gives consumer groups an important property:

Fault tolerance

If one consumer fails, the group can continue processing.

But What Exactly Is a Rebalance?

A rebalance happens when Kafka needs to redistribute partitions among consumers in a group.

This can happen when:

• A consumer joins the group
• A consumer leaves
• A consumer crashes
• Partitions are added to the topic
• Group membership changes

Imagine:

Before

C1 → P0, P1
C2 → P2, P3
C3 → P4, P5

C3 leaves.

Get Satyam’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Kafka needs to redistribute:

After

C1 → P0, P1, P4
C2 → P2, P3, P5

This process is called a rebalance.

In older Kafka consumer designs, frequent rebalances could cause noticeable pauses.

Modern Kafka has introduced mechanisms such as static membership and cooperative rebalancing to reduce unnecessary disruption.

But rebalancing is still something you need to understand when operating Kafka at scale.

One Topic, Multiple Consumer Groups

This is probably the most important mental model.

Suppose we have:

Kafka

orders
  │
  ├── payments-group
  ├── restaurant-group
  ├── delivery-group
  ├── notification-group
  └── analytics-group

Each group sees the topic independently.

So:

Order #101

can be consumed by:

Payments
Restaurant
Delivery
Notification
Analytics

That’s not five copies of the message stored separately for each application.

Kafka stores the message in the topic partitions and maintains consumer progress independently for each group.

This is one reason Kafka works well as an event backbone for large distributed systems.

Consumer Group vs Consumer

These two terms are often confused.

A consumer is an application instance that reads records from Kafka.

A consumer group is a collection of consumers working together.

Think of it like a company.

Company
   │
   ├── Employee 1
   ├── Employee 2
   └── Employee 3

Employees are consumers.

The company is the consumer group.

The company receives a workload and divides it among its employees.

If one employee leaves, the remaining employees can take over the workload.

Consumer Group vs Topic

Another common confusion:

A topic answers:

Where are the messages?

A consumer group answers:

Which logical application is processing them?

For example:

Topic
orders

contains events.

While:

payments-group

represents the Payment Service’s consumption of those events.

You can have:

orders
   │
   ├── payments-group
   ├── analytics-group
   └── delivery-group

Same topic.

Different consumers.

Independent progress.

A Very Important Production Scenario

Let’s say your analytics system suddenly becomes slow.

You have:

orders
   │
   ├── payments-group   ⚡
   ├── delivery-group   ⚡
   └── analytics-group  🐌

The analytics group is falling behind.

You might see its consumer lag increasing.

For example:

payments-group
Lag: 100

delivery-group
Lag: 50
analytics-group
Lag: 250,000

This doesn’t necessarily mean Kafka itself is slow.

It means:

The analytics consumers are not keeping up with the rate at which messages are being produced.

You could potentially scale the analytics consumers horizontally, assuming the topic has enough partitions to distribute the work.

For example:

Before

6 partitions
2 consumers
C1 → P0 P1 P2
C2 → P3 P4 P5

After scaling:

6 partitions
6 consumers

C1 → P0
C2 → P1
C3 → P2
C4 → P3
C5 → P4
C6 → P5

Now more consumers can process partitions in parallel.

This is one of the main reasons Kafka consumer groups are so useful for scaling.

A Common Mistake

People sometimes think:

“If I have 10 consumers, Kafka will process 10 things simultaneously.”

Not necessarily.

It depends on the number of partitions.

If you have:

3 partitions
10 consumers

you don’t get 10-way partition-level parallelism.

You effectively have:

C1 → P0
C2 → P1
C3 → P2

C4 → idle
C5 → idle
C6 → idle
...
C10 → idle

So when designing a Kafka system, you need to think about:

Producer throughput
        ↓
Partition count
        ↓
Consumer parallelism
        ↓
Consumer processing speed

These pieces are connected.

What Happens If I Have Two Different Consumer Groups?

Suppose:

orders topic

          ┌── payments-group
          │
          ├── analytics-group
          │
          └── delivery-group

Every group gets its own view of the stream.

For example:

Order #101

       ┌───────────────┐
       │ orders topic  │
       └───────┬───────┘
               │
      ┌────────┼────────┐
      ▼        ▼        ▼
 Payments   Analytics  Delivery

This is different from having multiple consumers inside the same group.

Inside one group:

P0 → C1
P1 → C2

The consumers share the work.

Across different groups:

P0 → Payments
P0 → Analytics
P0 → Delivery

Each group gets to process the event independently.

Remember this distinction.

It explains a huge part of Kafka’s architecture.

A Simple Rule I Use to Remember It

Here’s the mental model:

Same consumer group = divide the work

             Topic
               │
        ┌──────┼──────┐
        ▼      ▼      ▼
       C1     C2     C3

       Shared workload

Different consumer groups = repeat the work

              Topic
                │
       ┌────────┼────────┐
       ▼        ▼        ▼
    Group A   Group B   Group C

    Each group processes
    the events independently

That’s it.

Once this clicks, consumer groups become much easier to understand.

The 5 Things You Should Remember

If you’re preparing for a Kafka interview, remember these five points.

1. Consumer groups enable parallel processing

Multiple consumers in the same group can process different partitions simultaneously.

2. A partition is assigned to only one consumer in a group at a time

This is what allows Kafka to distribute work without multiple consumers in the same group normally processing the same partition simultaneously.

3. Different consumer groups consume independently

Payments and Analytics can both consume the same orders topic without sharing their offsets.

4. More consumers don’t automatically mean more throughput

Your parallelism is constrained by the number of partitions and the consumers’ ability to process records.

5. Consumer failures trigger rebalancing

When group membership changes, Kafka redistributes partitions among the remaining members.

The Bigger Picture

Consumer groups are not just a Kafka feature.

They are one of the ideas that make Kafka useful as a distributed event streaming platform.

The basic architecture is surprisingly simple:

               Kafka Topic
                     │
          ┌──────────┼──────────┐
          │          │          │
          ▼          ▼          ▼
       Group A    Group B    Group C
          │          │          │
       C1 C2 C3    C1 C2      C1 C2 C3

Each group represents a different logical application.

Consumers inside a group share the work.

Different groups independently consume the stream.

Once you understand that distinction, concepts like:

• Consumer Lag
• Rebalancing
• Partition Assignment
• Horizontal Scaling
• Offset Management
• Fault Tolerance

start making much more sense.

And that’s really the beauty of Kafka.

The APIs can look complicated.

The underlying idea doesn’t have to be.

Final Mental Model

If you remember only one diagram from this article, remember this:

                       KAFKA
                         │
                    orders topic
                         │
          ┌──────────────┼──────────────┐
          │              │              │
          ▼              ▼              ▼
      Payments       Analytics       Delivery
       Group           Group           Group
       │ │ │            │ │            │ │
       ▼ ▼ ▼            ▼ ▼            ▼ ▼
      C1 C2 C3         C1 C2          C1 C2

Same group → share the partitions.

Different groups → independently consume the topic.