Kafka Lag Wasn’t Going Down — Until We Changed the Partitioning

A few months ago, we suddenly saw a spike in Kafka consumer lag for one of our newly onboarded clients.

At first, it looked like a straightforward consumer-performance issue.

Messages were coming in faster than we could process them, and the backlog kept growing.

So we started investigating.

It Started With the Consumer

We found that some newly deployed code on the consumer side was taking longer than expected to process records.

The consumer was polling a batch of messages, processing them, and then coming back for the next batch.

The problem was that processing was taking longer than expected.

That meant the time between two poll() calls was also increasing.

Eventually, Kafka started considering the consumer unresponsive and triggered a rebalance.

So the situation started looking something like this:

Kafka
  |
  v
Consumer polls records
  |
  v
Process records
  |
  |---- Takes longer than expected
  |
  v
Next poll is delayed
  |
  v
Kafka considers consumer unresponsive
  |
  v
Rebalance
  |
  v
More processing delay
  |
  v
Lag increases

We had found one part of the problem.

Now we needed to reduce the pressure on the consumer.

First Attempt: Tune the Consumer

We changed two Kafka consumer configurations:

max.poll.interval.ms
max.poll.records

We increased max.poll.interval.ms to give the consumer more time to finish processing before Kafka considered it stuck.

At the same time, we reduced max.poll.records so that each poll returned fewer records.

The reasoning was simple:

Smaller batches + more time to process them = fewer unnecessary rebalances.

And it worked.

The lag started coming down.

That was a good sign.

So I thought, let’s give it another 1–2 hours and see where this goes.

Two hours later, the lag was still decreasing.

But there was a problem.

It wasn’t decreasing fast enough.

At that rate, we weren’t going to clear the backlog within the required timeframe.

So we needed more processing capacity.

The Next Move: More Consumers

I got my manager on a call and we discussed the situation.

The next obvious step was to increase the number of consumer pods and process messages in parallel.

But there was one important detail about our Kafka setup.

The topic had only one partition.

That meant adding more consumer pods wouldn’t give us the parallelism we were looking for.

With one partition, Kafka could assign that partition to only one consumer in the group at a time.

So this:

1 Topic
1 Partition
6 Consumer Pods

doesn’t mean:

6 consumers processing in parallel

It effectively means:

              Partition 0
                  |
                  v
             Consumer 1

             Consumer 2  → waiting
             Consumer 3  → waiting
             Consumer 4  → waiting
             Consumer 5  → waiting
             Consumer 6  → waiting

We needed a way to introduce more partitions without losing the existing backlog.

Get HRISHIKESH ZANJAD’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

And that’s where things got interesting.

Moving the Backlog to a New Topic

Instead of trying to work around the existing one-partition topic, we created a new topic with 6 partitions.

The old setup was:

Old Topic
    |
    v
Partition 0
    |
    v
Consumer

The new setup was:

New Topic
Partition 0 ──┐
Partition 1 ──┤
Partition 2 ──┤
Partition 3 ──┼──> Consumer Group
Partition 4 ──┤
Partition 5 ──┘

We then moved the existing backlog from the old topic to the new one.

Now we had something we didn’t have before:

actual partition-level parallelism.

But There Was One More Thing

We couldn’t just throw the messages into the new topic randomly.

Our producer was already using a Kafka key because ordering mattered for related messages.

Kafka’s ordering guarantee is within a partition.

The same key is consistently mapped to the same partition.

So the idea was:

Same key
   |
   +----> Same partition
   |
   +----> Same partition
   |
   +----> Same partition

While different keys could be distributed across different partitions:

Key A ──> Partition 1
Key B ──> Partition 4
Key C ──> Partition 2
Key D ──> Partition 5

This gave us the combination we needed:

parallelism across different keys, while maintaining ordering for messages belonging to the same key.

When moving the existing backlog to the new topic, we preserved the same keying strategy.

Then We Waited

At this point, there wasn’t much left to do.

We started monitoring the consumer lag partition by partition.

And finally, we saw what we wanted.

Partition 0 → Lag decreasing
Partition 1 → Lag decreasing
Partition 2 → Lag decreasing
Partition 3 → Lag decreasing
Partition 4 → Lag decreasing
Partition 5 → Lag decreasing

The backlog was now being processed concurrently.

The lag started dropping much faster.

And eventually, it came down as expected.

What I Learned From This

This incident reinforced one simple thing for me:

Before deciding what to scale, first identify the actual bottleneck.

We started by tuning the consumer, and that helped.

But when we needed more processing capacity, adding more pods alone wasn’t enough.

We had to change how the workload was distributed.

Moving the backlog to a topic with more partitions gave our consumers the parallelism they needed.

And because we were already using a Kafka key for ordering, we could preserve the same keying strategy while distributing the workload across partitions.

Final Takeaway

Kafka consumer scalability isn’t just about adding more pods.

You need enough partitions to actually make use of those consumers.

More consumers can give you more processing capacity only when there are enough partitions to distribute that work across.

What started as:

“Why is Kafka lag increasing?”

ended up becoming:

“Where is the actual limit to our parallelism?”

And in our case, the answer was:

The partition.