In-Depth Guide on Apache Kafka

In this article, we’ll explore Apache Kafka’s architecture and how it works, take a deep dive into topics, consumer groups, delivery guarantees, and schema registry to make a scalable and fault-tolerant messaging system. Also, get an overview of kSQL DB, Kafka Streams, and Kafka Connect.

In the digital age, businesses generate vast amounts of data that must be processed in real time. Traditional batch processing systems struggle to meet the demand for low-latency and high-throughput data streams. Apache Kafka was developed to address these challenges by providing a fault-tolerant, scalable, and distributed event-streaming platform.

Kafka is a distributed log. We can gather events from the outside world, and store them in that log, and they’re stored in scalable and replicated in a fault-tolerant way.

When your data is stored as a stream of events, the way you run computations over that data will look different. We’re used to this idea of “let me execute a DB query, get results, and do something with it”. Now, we’ll write stream processors that transform events one at a time, and if we want to aggregate events and create some kind of snapshot of staple then stream processors can do that too.

Kafka is widely used in various industries. Following is a non-exhaustive list of places or scenarios where Kafka is used:

• Real-time analytics
• Log aggregation
• Event-driven architectures
• Data pipelines
• Fraud detection

Kafka Cluster, Producer, and Consumer

Kafka enables systems to produce and consume events efficiently. Producers are applications that send data to Kafka, while consumers read and process it for tasks like reporting or updating dashboards.

Data sources communicating with producers

Producer communicating with Kafka cluster

Consumer consuming data from Kafka cluster

A Kafka cluster consists of multiple brokers (they could be servers, VMs, or containers), each running a Kafka process with its own storage. These brokers work together as a single unit. In fully managed cloud services, storage, retention, and other configurations are handled automatically.

One of Kafka’s key strengths is the decoupling of producers and consumers. Producers send data to topics, allowing consumers to read it independently. This flexibility ensures both can scale and fail without impacting each other. Even when using Kafka Streams or ksqlDB, producers and consumers are always at play behind the scenes.

Later on in this article, we’ll go more over producers and consumers.

Apache ZooKeeper in Kafka

Kafka relies on Apache ZooKeeper to manage consensus for the distributed state. It helps brokers agree on critical tasks like leader election, metadata storage, and cluster coordination.

Apache Kafka architecture

ZooKeeper handles metadata for configurations, access control lists (ACLs), broker failures, recovery, and data replication across the cluster. However, Kafka is transitioning to KRaft (Kafka Raft metadata mode), which replaces ZooKeeper to make Kafka more self-contained and efficient.

Kafka Topics and Partitions

A topic is a logical grouping of related messages or events in Kafka. There is no theoretical limit to the number of topics, though the number of partitions per topic is practically limited.

Topics are divided into partitions, which are distributed across brokers for scalability. Each partition is an immutable log, meaning messages are strictly ordered within a partition but not across the entire topic. You can only append messages at the end, and the order cannot be changed.

Topics, partitions, and segments

Within a broker, each partition consists of multiple log segments — files stored on disk with associated indexes. While Kafka automatically distributes partitions across brokers, it does not rebalance them based on load. In a managed Kafka service, this is handled for you.

Partitions spread across brokers

Compacted Topics

If you need to store only the latest value for each key, you can use a compacted topic. For example, when tracking device metrics by device ID, compacted topics retain only the most recent entry for each key, discarding older values.

Compacting Topics

Kafka Log

A log in Kafka is an immutable sequence of events, where new messages can only be appended to the end. The order of entries represents the sequence of events over time.

Kafka Log

Each log entry has an offset, a unique number that starts at 0 and increases monotonically. Every partition has its own offset space, meaning multiple consumers can read the same partition independently without affecting each other. This enables real-time processing and delayed reprocessing as needed.

In Kafka, a log is also referred to as a stream, and each entry is an event. Events are stored as key-value pairs with a timestamp (creation or ingestion time, defaulting to Kafka’s system time if not provided) and optional headers for metadata.

Data Retention Policy

Kafka allows you to configure data retention based on policies, compliance requirements (e.g., GDPR), or storage costs. By default, data is retained for one week, but this can be adjusted globally or per topic as needed.

Each partition consists of segments that store data. When the latest record in a segment exceeds the retention period, the segment is automatically deleted.

Kafka Brokers: Replication and Load Balancing

A broker in Kafka is responsible for managing partitions, storing log files, handling producer inputs, processing consumer requests, and facilitating pub/sub communication.

Partition Leadership and Replication

Kafka ensures fault tolerance by replicating partitions across multiple brokers. Each partition has a leader and one or more follower replicas. All writes from producers and reads from consumers go through the leader, while followers stay synchronized by replicating data.

Partition leader and followers

From Kafka version 2.5 onward, consumers can also read from replicas. This is beneficial when replicas are distributed across different availability zones, reducing latency by serving consumers from the nearest replica.

A Kafka cluster can have n brokers and m partitions for a given topic, with no direct correlation between them.

If a broker hosting a leader partition fails, one of its replicas on another broker is automatically elected as the new leader. This ensures continuous availability and prevents data loss without manual intervention.

Load Balancing and Partitioning

When a producer sends data to a topic, it actually writes to a partition, which is stored on a specific broker. Kafka determines which partition to use based on:

• Round-robin (default for messages without a key): Ensures even distribution but doesn’t maintain order.
• Key-based hashing (hash(key) % number_of_partitions): Guarantees that messages with the same key always go to the same partition, preserving order.

For custom logic, developers can implement a custom partitioner, though this is rarely needed.

Consumer Groups

Consumers pull data from one or more topics in a Kafka cluster. They track the last read message using an offset, which is stored both in memory and in a special Kafka topic called consumer offset. This allows consumers to resume reading from where they left off, even after restarts.

Multiple applications can consume the same data, and all consumers belong to a consumer group, even if there is only one consumer in the group. You can scale consumers by adding more instances to a consumer group. For example, in Kubernetes, you can deploy multiple containers running the same consumer application, all within a single consumer group.

Consumer Groups

In a consumer group, partitions are automatically assigned to consumers fairly and cooperatively. If consumers are added or removed, Kafka rebalances the partition assignments dynamically.

However, the Consumer API does not handle state migration. Consumers are often stateful, meaning they track previously read messages or perform aggregations. If a consumer instance fails, its state is lost. To solve this, Kafka Streams and kSQL DB provide built-in state management, ensuring fault tolerance and elasticity.

More on Producers

Producers can be implemented in Python, Java (best suited since Kafka is written in Java), Go, C/C++, and other languages using native or community-built packages. If a language isn’t supported, the REST Proxy can be used. A CLI tool is also available for testing.

Get AkashSDas’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

When a producer calls the send method to publish data to Kafka, several processes happen behind the scenes:

1. Serialization — The data is converted into bytes based on a configured serializer.
2. Partitioning — The partitioner determines which partition the data should go to, based on whether a key is provided.
3. Buffering & Batching — Data is temporarily buffered before being sent over the network to a broker. The balance between batch size and flush frequency impacts latency vs. throughput, and these settings can be tuned.

Producer Guarantees

After sending data, the producer waits for an acknowledgment (configurable with the acks setting):

• Acks 0 (NONE): No acknowledgment is required. Data is sent over the TCP socket, but there’s a risk of data loss if failures occur. Lowest latency.
• Acks 1 (LEADER): The leader broker writes the data to disk and sends an acknowledgment. Replicas may not have the data yet. Moderate latency.
• Acks -1 (ALL): The leader writes the data to disk, and all replicas must sync before acknowledgment. Ensures reliability but increases latency.

How Exactly Once Works

In distributed systems, true exactly-once delivery is impossible due to atomic broadcast limitations. Messages may be delivered more than once, but Kafka’s Exactly Once Semantics (EOS) ensures each message is processed only once, even if duplicates exist.

Kafka achieves this via:

• Idempotent producers, which detect and discard duplicates.
• Transactional APIs, ensuring atomic writes across partitions.

Implementing this manually in producers and consumers is feasible but complex. Kafka Streams and kSQL DB simplify it with built-in configurations.

Security

Kafka provides encryption, authentication, and authorization to secure data transmission and access.

• Encryption in Transit: Kafka encrypts data moving between clients, brokers, and ZooKeeper using TLS.
• No Built-in Encryption at Rest: Kafka does not encrypt stored data in partitions by default. External solutions must be used for this.
• Authentication & Authorization: Kafka supports user authentication and access control lists (ACLs) to manage permissions within the cluster.

Apache Kafka Connect

Kafka Connect is a distributed, fault-tolerant, and pluggable data integration framework that simplifies moving data into and out of Kafka.

Instead of writing custom producers and consumers to connect with external systems (like databases, S3, or messaging platforms), Kafka Connect provides ready-to-use connectors for common data sources (sources) and destinations (sinks).

Examples of Source Connectors (bringing data into Kafka):

• Amazon Kinesis
• RabbitMQ, MQTT
• InfluxDB
• Salesforce
• JDBC

Examples of Sink Connectors (writing data from Kafka):

• Elasticsearch
• Hadoop
• Amazon S3
• Neo4j
• InfluxDB
• Cassandra

How Kafka Connect Works

Kafka Connect runs on a dedicated infrastructure with a cluster of Connect workers. Each worker runs multiple tasks, which execute pieces of source or sink connectors.

• Parallelism and Scaling: Connectors can be partitioned across multiple workers for better performance. Some connectors (like database readers) may run as a single task, while others (like writing to Elasticsearch) can be split across multiple workers.
• Failover and State Management: Kafka Connect automatically tracks data offsets and stores state information in internal Kafka topics, ensuring reliability and fault tolerance.

Schema Registry

As data evolves, different systems may need to work with different schema versions. A Schema Registry solves this by managing and validating schemas for data written to or read from Kafka. Confluent provides a widely used Confluent Schema Registry for this purpose.

Schema Registry allows you to store schemas (commonly in AVRO format) and enforce compatibility rules at runtime.

Example AVRO Schema:

{
  "type": "record",
  "name": "User",
  "fields": [
    {"name": "id", "type": "int"},
    {"name": "name", "type": "string"}
  ]
}

Schemas can also be versioned and tracked in Git, ensuring better change management over time.

More on Processing Data

Processing data in Kafka can get complex, especially with operations like aggregation, mapping, and reduction. Additionally, Kafka doesn’t inherently track processing state or progress, making failure handling difficult. To address this, kSQL DB and Kafka Streams are used.

Kafka offers three ways to process streams, ranked from simplicity to flexibility:

1. kSQL DB — Easiest, SQL-based.
2. Kafka Streams API — More control, Java-based.
3. Custom Producers/Consumers — Most flexible but complex.

kSQL DB

kSQL DB is a SQL-based streaming database that allows processing Kafka topics using SQL queries instead of programming in Java, Python, etc.

• Queries run continuously on new data and produce results into a topic.
• It operates on a cluster of kSQL DB servers adjacent to the Kafka cluster.
• The REST API is used to submit queries.
• Offers the full power of Kafka Streams API without requiring code.

Kafka Streams

Kafka Streams is a Java API built into Apache Kafka for stream processing.

• Enables complex processing with stateful logic.
• Manages state, failures, scalability, and fault tolerance efficiently.
• Supports exactly-once processing semantics for reliable data handling.

Unlike Kafka clusters or consumer groups, it is just a library that integrates into existing applications.