Spring Kafka Tutorial

Apache Kafka

If you are here, most probably you already know what Kafka is. I will give a very small introduction, and move quickly to my agenda for this blog(maybe it will be in parts, not sure yet). This tutorial will be fast paced at some places, and slow at other. The goal of this is to give you an extensive hands-on to quickly get you started on whatever project you are trying to integrate spring-kafka to.

I believe, the biggest problem with the other tutorials out there is, they are walking you through a project they have already made, skipping the hard part of facing issues while setup, and other prerequisites we need for our projects to be production ready. Through this, I am building this project myself, from scratch, and I will try to walk you through the process.

Github Source for the Blog: https://github.com/Geek8080/notification-service

Introduction

Apache Kafka is an open-source distributed streaming platform used for high-performance data pipelines, streaming analytics, data integration, and mission-critical applications.

Confused? Don’t be. The above definition doesn’t matter for now. We are here to learn it, not just remember facts.

To understand what kafka is, think of it like, sensors in car. Imagine you have a bunch of sensors in your car that are constantly collecting data about speed, engine temperature, and fuel level. Kafka would be the system that takes all that data and streams it to different parts of your car, like the dashboard, the fuel gauge, and even the engine itself, so that everything can work together seamlessly.

If you are not interested in learning the history(which I highly suggest to know), skip to the next blog, where we setup our spring maven project.

History and Creation of Kafka

Want a real life scenario where Kafka is used? Let’s dive into a little history, what was the first use case, the problem which Kafka solved.

Kafka was created by engineers at LinkedIn. The creation of Kafka at LinkedIn is a fascinating story fueled by the need for real-time data processing.

In the late 2000s, LinkedIn was experiencing explosive growth. Imagine this bustling marketplace of professional connections, with millions of users signing up, updating profiles, and interacting with each other every day. This generated a tsunami of data, including:

• User activity: Profile views, connection requests, endorsements, posts, etc.
• Website interactions: Clicks, searches, page views, etc.
• Infrastructure logs: Server activity, system events, error messages, etc.

The problem was, existing infrastructure couldn’t handle it.

• Batch processing: Traditional data analysis involved waiting until the end of the day to crunch all the data from the previous 24 hours. This was too slow for a platform that needed to react in real-time. Imagine waiting until tomorrow to know which recommendations to show a user or which suspicious activity to flag!
• Messaging solutions: Existing enterprise messaging systems weren’t built for the volume and variety of data LinkedIn was generating. It was like trying to fit a giant elephant into a tiny car!

This data bottleneck was throttling LinkedIn’s growth and potential. They needed a new way to ingest, process, and analyze data in real-time, and that’s where the story of creation of Kafka begins.

Engineers at LinkedIn, set out to solve this data dilemma. They envisioned a platform that could:

• Ingest data in real-time: No more waiting for overnight batch processing!
• Scale effortlessly: Handle ever-growing volumes of data without breaking a sweat.
• Be flexible: Adapt to a diverse range of data types and use cases.

After months of brainstorming and coding, Kafka was born! Named after the famous author Franz Kafka, whose work is known for its complex and labyrinthine structures, the platform mirrored the intricate flow of data within LinkedIn.

Get Prashantprakash’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Here’s how it worked:

• Data producers: Applications sent streams of data (think user updates, website clicks) to Kafka topics, like dedicated channels for specific data types.
• Kafka brokers: These acted as data hubs, storing and distributing the messages across a network of computers.
• Data consumers: Other applications subscribed to specific topics and received data in real-time, allowing for immediate processing and analysis.

Kafka revolutionized LinkedIn’s data processing landscape. It enabled:

• Real-time personalization: Tailored recommendations and notifications based on user activities.
• Fraud detection: Quickly identify and stop suspicious activity.
• Streamlined analytics: Gain valuable insights from constantly flowing data.

The next part is about to get technical, enough with stories. I will keep it short, you can go here to read about these in detail, or you can just follow this blog, I will walk you through all the terminologies.

Important Basic Terminologies

Any system with Kafka integration will have three parts:

1. Producer: The part of system which produces event(data) to Kafka server.
2. Kafka Server: The Kafka server itself, storing the data(there’s a lot more to the functionality).
3. Consumer: The part of system which consumes event(data) from the Kafka server.

To define event, Events are the individual pieces of data, like sensor readings, stock quotes, or website clicks, flowing through the system.

Events on a Kafka server are organized according to the topic. Whenever we produce any data to Kafka, we attach the topic name to it. It defines the kind of data event is holding. Like if we push data related to post clicks, we can simply push it to a topic called post-clicks. This will also facilitate the consumer to consume similar data at once.

To handle large volumes of data efficiently, Kafka divides topics into partitions. Think of these as smaller conveyor belts within the main topic. Spreading events across partitions allows multiple consumers to process them simultaneously, boosting performance and scalability.

We will learn later, we can choose a particular partition to consume data from.

The Kafka server is run in clusters. Each individual server in a Kafka Cluster is called a Broker. Since there are multiple brokers, there needs to be someone, to look after them, make sure they are configured right and working fine. This is done by Zookeeper. So, we can say, Kafka server is made up of Kafka Cluster and Zookeeper.

We often maintain replica of data on Kafka Cluster. Data replication in Kafka ensures fault tolerance, high availability, and improved read performance by storing copies across multiple brokers.

Note that, topic can be partitioned across different brokers.

For following this tutorial trail, download the following:

1. Any Java IDE of your choice
2. Java 17 or higher
3. Maven 3 or higher
4. Apache Kafka: https://kafka.apache.org/downloads
5. Offset Explorer: https://kafkatool.com/download.html

Blogs in this tutorial trail:

1. https://medium.com/@prashantprakash97/spring-kafka-tutorial-setting-up-project-d979ea0ae26a
2. https://medium.com/@prashantprakash97/spring-multi-module-maven-project-51802214227e
3. https://medium.com/@prashantprakash97/spring-boot-logging-implementation-in-a-multi-module-project-21f62c53f51f
4. https://medium.com/@prashantprakash97/spring-kafka-producer-part-1-api-and-producer-architecture-6f0f036c9a64