Apache Kafka — Topics, partitions, and offsets

Kafka topics are used to store and organize events in a Kafka cluster. A topic is a stream of data within the cluster, and a Kafka cluster can have many topics. Each topic is identified by its name, which must be unique across the entire Kafka cluster. For example, a topic could be named “payments,” “orders,” or “logs.

A producer sends a message to a Kafka topic, and a consumer reads data from the topic. Topics in Kafka are multi-producer and multi-subscriber, meaning that a topic can have one or more producers that write data to it and one or more consumers that read from it.

A topic is similar to a table in a database, but without any constraints. Producers can send data to a topic in any format, and Kafka topics support any kind of message format, such as JSON, text files, binary files, or anything else. The sequence of messages in a topic is called a data stream, and messages in a topic can be read as often as needed. By default, Kafka retains all published messages for a limited amount of time, whether or not they have been consumed. The default retention time is 168 hours (i.e., one week), but this can be configured in the configuration file (using the log.retention.hours setting).

In a Kafka cluster, topics can be divided into multiple partitions. A topic can have one or more partitions, and these partitions are located on different Kafka brokers. The placement of data is important for scalability, as it allows clients to read and write data from many brokers at the same time. The number of partitions in a topic is defined at the time of topic creation, and they are numbered 0, 1, 2, and so on.

When a producer publishes a message to a Kafka topic, it is appended to one of the topic’s partitions. If a message includes a message key (such as a user ID or order ID), it will always be written to the same partition. A message key can be a string, number, object, or anything else, but it is optional; if no key is provided, the key will be set to null and the message will be sent in a round-robin fashion to different partitions. Kafka uses a key-hashing mechanism to distribute messages with keys to the appropriate partitions.

Once a message is written to a Kafka topic, it cannot be changed. This is called immutability, and it means that Kafka topics are immutable.

Offsets

In Kafka, each message within a partition is assigned a unique sequential integer called an “offset.” The offset is used to identify each message within the partition. By default, the offset starts at 0 for each partition and is incremented by 1 each time a new message is received. However, offsets are specific to a partition, so the message with offset 1 in partition 0 may not be the same as the message with offset 1 in partition 1.

Kafka guarantees the order of messages within a partition, but there is no guarantee of order between messages in different partitions of the same topic. Additionally, Kafka messages are deleted over time, so once a message is deleted, its offset will no longer be used. Offsets in Kafka are continuously increasing in a never-ending sequence.

Kafka is designed to be fault-tolerant and highly resilient through data replication. In Kafka, topics can be replicated across multiple brokers, meaning that duplicate copies of the data are written to multiple servers. This prevents data loss in case of failure and ensures that the data is always available for consumption. Replication in Kafka is performed at the partition level, and the replication factor (i.e., the number of copies of the data) can be set at the time of topic creation. A replication factor of 3 is commonly used in production environments, which means that there will always be three copies of the data. A replication factor of 1 means that there is no replication.

Kafka Topic CLI

List Kafka topics— To list all Kafka topics in the Kaka server, We will use the –list command.

For mac and Linux, use kafak-topic.sh command, for windows, you can use the kafka-topic.bat command. The — bootstrap-server represents the Kafka broker we’re trying to connect to. My Kafka broker is launched locally, therefore, I use localhost:9092.

➜  ~ kafka-topics --bootstrap-server localhost:9092 --list

It returns test_topic because I have only one topic created on my Kafka server

Create a Kafka topic — You can create a topic on the Kafka server by using the –create command. See the example below

➜  ~ ~ kafka-topics --bootstrap-server localhost:9092 --topic my.first.topic --create

Our first topic has been created with only one partition, and if we do not specify any configuration at the time of its creation, it will use the default settings.

As you can see we get a warning because I am using a dot. It simply tells us not to use (.) and (_) with the same topic name. For example — If I try to create a topic with the name my_first_topic, it will not allow me to create it because it will collide with my.first.topic name.

As we can see in the above image, Kafka throws an exception when we try to create a topic with the name my_first_topic. It clearly says my_first_topic collides with existing topic my.first.topic. We cannot use both underscore and dots interchangeably for the same topic name.

The topic name must consist only of ASCII alphanumeric characters and the characters ‘.’, ‘_’, and ‘-’.

Create a Topic with partitions

We will use the — partitions command to create a topic with partitions. In this example, We will create a topic with 3 partitions

➜  ~ kafka-topics --bootstrap-server localhost:9092 --topic my.second.topic --create --partitions 3

my.second.topic has been created with 3 partitions.

Note — The replication factor of both of these topics is one by default as we have not specified any configuration for the replication factor.

Get Shweta Tyagi’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Let’s create a topic with a replication factor. To create a topic with a replication factor, we need to specify the — replication-factor option in create topic command.

➜  ~ kafka-topics --bootstrap-server localhost:9092 --create --topic my_topic --partitions 3 --replication-factor 2

In the above example, I tried to create a topic with a replication factor of 2. But as you can see, I got an error that says Replication factor: 2 larger than available brokers: 1.

I got this error because I have only one broker in my Kafka cluster which is less than the given replication factor. So, if you try to create a topic with a replication factor greater than the number of brokers in your Kafka cluster, this will not work. You can create a topic with a replication factor, less than or equal to the number of brokers in your Kafka cluster. But, in the future, if you have a Kafka cluster that has more brokers, then you can define a higher replication factor for your topics.

So, Let’s create a topic with only one replication factor

Now, my_topic has been created without any error :)

Describe a Topic

To describe a Kafka topic and get partition details, we need to provide the –describe option.

➜  ~ kafka-topics --bootstrap-server localhost:9092 --describe --topics my_topic

We can see from the above image, my_topic has 3 partitions with a replication factor of 1. Partition 0 is on broker leader 0, and partition 1 is also on broker leader 0.

If you don’t specify –topic in the above command, it will describe all the topics in that cluster. See the example below

➜  ~ kafka-topics --bootstrap-server localhost:9092 --describe

You can see in the above pic, it describes all the topics present in the server

Modify a Topic

To increase the number of partitions in Kafka topics, use the –alter command. Let’s increase the number of partitions in my.first.topic to 3.

➜  ~ kafka-topics --bootstrap-server localhost:9092 --alter --topic my.first.topic --partitions 3

— alter command does not have any output. You can use the — describe command to verify the results.

By using this command, you can only increase the number of partitions, you can not remove them. This is a dangerous command and is not recommended if your application is using key-based ordering as increasing the number of partitions can impact the key hashing technique.

Delete a Topic

To delete a topic, we will use the –delete command with the topic name which we want to delete. Below is the example

➜  ~ kafka-topics --bootstrap-server localhost:9092 --delete --topic my.first.topic

–delete command does not have any output. To verify if a topic has been deleted or not, you can use either the –list or –describe command. You can see from the output of –list command that my.first.topic has been deleted successfully.

To delete more than one topic, you can pass the comma-separated list of topics to the –delete command. See below example

➜  ~ kafka-topics --bootstrap-server localhost:9092 --delete --topic my.first.topic, my.second.topic

It will delete both my.first.topic and my.second.topic topic.

Deletion of a topic depends on broker settings(delete.topic.enable), if it is not enabled, the topic will not be deleted. It will be marked for deletion but will not be deleted.

For Apache Kafka installation, you can refer to my previous article on Kafka https://medium.com/devops-dev/apache-kafka-getting-started-with-kafka-61948e50d1ab

References:

https://kafka.apache.org/quickstart