Kafka for Beginners: Topics, Partitions, Murmur2, ZooKeeper, KRaft, Replication and Acknowledgements — Part2

Continuing from our previous part, where we discussed basics of Kafka and it’s component. Now it’s time to dive into partitioning and how it works in Kafka.

1. Now Let’s Return to Partitions

Suppose we have:

Topic: orders

P0
P1
P2

A producer sends:

Key   = customer-101
Value = Order-5001

The producer needs to decide:

Which partition should receive this record?

This is where the partitioner comes in.

2. How Does Kafka Select a Partition?

The Kafka producer has a component called the partitioner.

The partitioner determines which partition should receive a record.

There are two important scenarios:

1. The record has a key.
2. The record does not have a key.

Let’s first look at a keyed record.

3. Partition Selection When a Key Is Present

Suppose our topic has three partitions:

Topic: orders

P0
P1
P2

The producer sends:

Key   = customer-101
Value = Order-5001

Conceptually:

customer-101
                   |
                   v
                Hashing
                   |
                   v
              Hash value
                   |
                   v
              Partitioner
                   |
                   v
                Partition

A simplified view is:

partition = hash(key) % number_of_partitions

The actual partition selection is performed by the configured producer partitioner.

4. What is Murmur2?

Murmur2 is a hashing algorithm.

In the traditional/default Kafka producer behavior for keyed records, the key is passed through a Murmur2 hash, and that hash participates in selecting the partition.

Conceptually:

Record Key
    |
    v
Murmur2
    |
    v
Hash Value
    |
    v
Partitioner
    |
    v
Partition

For example:

customer-101
      |
      v
   Murmur2
      |
      v
 Hash value
      |
      v
Partition 1

It is important to distinguish:

Murmur2
   =
Hashing algorithm

Partitioner
   =
Component responsible for partition selection

So the overall view is:

Key
 |
 v
Murmur2
 |
 v
Hash
 |
 v
Partitioner
 |
 v
Partition

5. Same Key Normally Goes to the Same Partition

Suppose we send:

customer-101 → Order A
customer-101 → Order B
customer-101 → Order C

The same key goes through the same partitioning logic.

Therefore, under unchanged partitioning conditions:

Partition 1

Order A
Order B
Order C

This is one of the most important reasons to use a Kafka key.

Get Ankur Rastogi’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

It allows records for the same entity to remain ordered within a partition.

6. Different Keys Can Go to Different Partitions

Suppose:

customer-101
customer-102
customer-103

The partitioner processes each key.

Conceptually:

customer-101 → P0
customer-102 → P2
customer-103 → P1

So:

Partition 0
-----------
customer-101

Partition 1
-----------
customer-103

Partition 2
-----------
customer-102

This allows different keys to be processed in parallel.

7. What Happens When There Is No Key?

Suppose:

Key = null

There is no key to hash.

Modern Kafka producer clients use sticky partitioning behavior for records without keys.

The producer can send records to a partition as a batch and later switch to another partition.

Conceptually:

Order A → P1
Order B → P1
Order C → P1

then

Order D → P2
Order E → P2

Therefore:

Without a key, you should not assume that related records will always go to the same partition.

8. Increasing the Number of Partitions

Suppose we initially have:

P0
P1
P2

and:

customer-101 → P1

Now we increase the topic to six partitions:

P0
P1
P2
P3
P4
P5

The partition calculation is now based on a different number of partitions.

Before:

hash(key) % 3 → P1

After:

hash(key) % 6 → potentially P4

Therefore:

Increasing the number of partitions can change the partition assignment of keyed records.

This is important for applications that depend on key-based ordering.

10. Partition vs Replication

Partitioning and replication solve different problems.

Partitioning

Provides:

• Scalability
• Parallelism
• Ordering within a partition

We will cover more on replication in next part .. Thanks for your valuable time.