A Step-by-Step Guide to Apache Kafka Installation and Configuration for Spring Boot Applications (Especially in Windows Machines)

Introduction:

Apache Kafka has become a pivotal technology in the world of distributed systems, providing a robust platform for building real-time data pipelines and streaming applications. In this article, I’ll walk through the process of installing and configuring Apache Kafka for local development (especially for Windows Machines), setting up Spring Boot applications as the producer and consumer, and demonstrating the communication between them.

Downloading, Installing and Executing few Apache Kafka commands on a Windows Machine:

1. Begin by downloading Apache Kafka from the official website: Click Here to download
2. Follow the commands from the Apache Kafka QuickStart page along with the instructions below.
3. Command to extract the Apache Kafka downloads and navigate to the extracted folder works perfectly fine on Windows Machines.
4. Other commands specified in the Apache Kafka QuickStart page work fine for Linux and Mac machines.
5. To execute those commands on a Windows machine, we have to tweak the commands a bit. Please refer to the below image and the next steps!

Kafka folder after extracting

6. Once the Apache Kafka downloads are extracted, the Kafka/bin folder looks something like the above image.

7. Now according to the Apache Kafka QuickStart page, I have listed out the equivalent commands that can be used to execute on a Windows Machine.

Commands (For Windows Machine) according to the Apache Kafka QuickStart page

Note: The .sh files mentioned in the Apache Kafka QuickStart page work only on Linux and Mac machines.

Note: To execute commands on a Windows Machine, we have to use the .bat files present in the Kafka/bin/windows folder.

Note: Avoid spaces in the folder names. It will lead to issues while executing commands in the terminal.
Avoid this — D:\Software\Apache Kafka\kafka\bin
Use something like this — D:\Software\Kafka\kafka\bin

Open the terminal from the bin folder and execute the below commands.

Commands:

• Start Zookeeper Server — .\bin\windows\zookeeper-server-start.bat config\zookeeper.properties
• Start Kafka Server — .\bin\windows\kafka-server-start.bat config\server.properties
• Create a Kafka topic — .\bin\windows\kafka-topics.bat --create --topic quickstart-events --bootstrap-server localhost:9092
• Describe a Kafka topic — .\bin\windows\kafka-topics.bat --describe --topic quickstart-events --bootstrap-server localhost:9092
• Write events to Kafka topic — .\bin\windows\kafka-console-producer.bat --topic quickstart-events --bootstrap-server localhost:9092
• Read events from Kafka topic — .\bin\windows\kafka-console-consumer.bat --topic quickstart-events --from-beginning --bootstrap-server localhost:9092

The above are a few sample commands. Similarly, any other commands can be executed on a Windows Machine.

Get Harishankar Bhat R’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

The important thing here is to navigate to the Kafka/bin/windows folder and use the .bat files while on a Windows Machine.

Now we have downloaded, and installed Apache Kafka and created topics, written and read events to/from the topic, etc.

Let’s integrate Apache Kafka with Spring Boot applications

We need 2 Spring Boot applications. One as a Producer and another one as a Consumer.

In the Producer application, we can have a config class to configure the Apache Kafka topic like below.

Kafka Configuration

Once the Kafka topic is configured, we can use the below code to publish or write events to the Apache Kafka.

Code Snippet to Write or Publish events to Apache Kafka

A few handy configurations for the Producer application are below!!

Configuration related to Apache Kafka in the Producer application

Let’s see the configuration and code snippets required for the Consumer application.

The below code is used to listen to/consume messages/events from Apache Kafka.

Code snippet to consume Apache Kafka messages/events

A few handy configurations for the Consumer application are below!!

Configuration related to Apache Kafka in the Consumer application

Conclusion

By following the steps in this guide, you should now have a fully functional Apache Kafka setup with Spring Boot applications acting as the producer and consumer. This configuration allows for seamless communication between different components in a distributed system, opening the door to scalable and real-time data processing and streaming use cases. Feel free to explore further Kafka features and functionalities to enhance your streaming applications. Happy coding!

I would be grateful if you could share your feedback/suggestions on this article. Thank you in advance!!

I hope this article helps you in getting started with Apache Kafka.
See you soon in the next article!!