Kafka Architecture Fundamentals

What Actually Happens Inside Kafka?

Kafka is often introduced as:

“A distributed messaging system.”

That description is correct — but incomplete.

At an architectural level, Apache Kafka is better understood as a distributed event streaming platform designed to move, store, and process large volumes of events reliably.

If you are designing microservices, event-driven systems, payment platforms, telecom systems, or real-time data pipelines, understanding Kafka’s internal architecture is far more important than simply knowing how to publish and consume a message.

Let’s build the architecture from the ground up.

1. The Problem Kafka Solves

Imagine a system with several services:

                 ┌──────────────┐
                 │ Order Service│
                 └──────┬───────┘
                        │
                        ▼
                 ┌──────────────┐
                 │PaymentService│
                 └──────┬───────┘
                        │
                        ▼
                 ┌──────────────┐
                 │Notification  │
                 │   Service    │
                 └──────────────┘

Initially, services may communicate directly using REST or gRPC.

But as the system grows, synchronous communication creates dependencies.

For example:

Order
  │
  ├──► Payment
  │
  ├──► Inventory
  │
  ├──► Notification
  │
  └──► Analytics

Now ask:

• How do we retain events for later processing?
• What happens if Payment is unavailable?
• Should Order wait for Notification?
• How do we retry failed communication?
• How do we handle traffic spikes?
• How do multiple consumers receive the same business event?
• How do we retain events for later processing?

This is where event-driven architecture becomes useful.

Instead of directly calling every service:

Order Service
      │
      ▼
    Kafka
   /  |   \
  ▼   ▼    ▼
Payment Inventory Analytics

The producer publishes an event. Consumers decide independently when and how to process it.

This creates temporal decoupling between services.

2. Kafka’s Core Architecture

Apache Kafka Architecture

At a high level:

There are five concepts you should understand first:

1. Broker
2. Topic
3. Partition
4. Record
5. Consumer

Everything else builds on these.

3. What Is a Kafka Broker?

Kafka Brokers

A Kafka broker is a Kafka server responsible for storing and serving data.

A Kafka cluster typically contains multiple brokers. The purpose of multiple brokers is not simply performance.

They provide the foundation for:

• scalability
• replication
• fault tolerance
• distributed storage

Kafka distributes topic partitions across these brokers.

For example:

Broker 1       Broker 2       Broker 3
P0             P1             P2

A topic can therefore span multiple machines.

4. What Is a Topic?

A topic is a logical category or stream of events.

For example:

order-events
payment-events
customer-events
inventory-events
notification-events

A producer publishes records to a topic.

For example:

Order Service
     │
     │ OrderCreated
     ▼
  "order-events"

But an important architectural point is:

A topic is not where Kafka’s scalability comes from.

The scalability comes primarily from partitions.

5. What Is a Partition?

A topic is divided into one or more partitions.

For example:

Topic: order-events
Partition 0
────────────────────────
0 → Order A
1 → Order D
2 → Order G
3 → Order J

Partition 1
────────────────────────
0 → Order B
1 → Order E
2 → Order H

Partition 2
────────────────────────
0 → Order C
1 → Order F
2 → Order I

Each partition is an ordered, append-only log. This is one of the most important concepts in Kafka. Kafka does not continuously modify records inside a partition.

New records are appended to the end.

Partition 0
────────────────────────
0 → Order A
1 → Order D
2 → Order G
3 → Order J
4 → Event M
              ▲
              │
           append

This design is one of the reasons Kafka can achieve high throughput.

6. Why Partitions Matter

Suppose you have:

order-events

with only one partition.

Processing is fundamentally limited by that single partition’s ordering and throughput characteristics.

Now create:

order-events
P0
P1
P2
P3

Different consumers can process different partitions in parallel.

order-events
          ┌────┼────┐
          ▼    ▼    ▼
         P0   P1   P2
          │    │    │
          ▼    ▼    ▼
         C1   C2   C3

This is the fundamental mechanism behind Kafka’s horizontal scalability.

More partitions can provide more parallelism. But there is an important trade-off:

Partitions are not free.

More partitions mean more operational overhead, more metadata, more files, more replication work and potentially more expensive rebalancing.

So partition count is an architecture decision, not just a performance setting.

7. What Is a Kafka Record?

A producer doesn’t technically send a “message” in the architectural sense.

It produces a record. A record can contain:

Key
Value
Headers
Timestamp

For example:

Key:
customer-123

Value:
{
  "orderId": "ORD-1001",
  "amount": 5000,
  "currency": "INR"
}
Headers:
eventType = OrderCreated
traceId   = abc-123

The key becomes particularly important when partitioning is involved.

8. How Does Kafka Decide Which Partition Receives a Record?

Consider:

Topic: order-events
P0
P1
P2
P3

If a producer sends a record with a key:

customer-123

the producer’s partitioning logic determines which partition receives that record.

Conceptually:

customer-123
      │
      ▼
Partitioning logic
      │
      ▼
     P2

If the same key consistently maps to the same partition, events for that key can maintain ordering.

For example:

customer-123

OrderCreated
PaymentCompleted
OrderShipped
OrderDelivered

can be placed in the same partition.

That gives us:

Ordering per key within a partition.

Not global ordering across the entire topic.

This distinction becomes extremely important when designing distributed systems.

9. The Most Important Kafka Ordering Rule

Kafka guarantees ordering within a partition. It does not guarantee ordering across partitions.

Suppose:

P0:
OrderCreated
PaymentCompleted

P1:
OrderShipped

Kafka does not provide a global sequence such as:

OrderCreated
→ PaymentCompleted
→ OrderShipped

across P0 and P1.

Therefore, when ordering matters, the architect must decide:

What is my ordering key?

For an order system, it might be: orderId

For a customer system: customerId

For a bank account: accountId

This is an architectural decision.

10. What Is an Offset?

Every record inside a partition has an offset.

For example:

Partition 0
Offset    Event
0         OrderCreated
1         PaymentInitiated
2         PaymentCompleted
3         OrderShipped
4         OrderDelivered

The offset identifies the record’s position within that partition.

Notice:

P0 → offset 0,1,2,3...
P1 → offset 0,1,2,3...

Offsets are therefore partition-specific. There is no single global offset for a topic.

11. Why Offsets Matter

Offsets allow consumers to track their position. Imagine a consumer has processed:

0
1
2

and is currently processing:

3

If the consumer crashes, Kafka can use the consumer group’s committed offset to determine where processing should resume.

Get Malti Yadav’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Conceptually:

Partition
0 ✓
1 ✓
2 ✓
3 ← processing
4
5

After restart:

Resume from committed position

This is fundamentally different from a traditional queue where a message may disappear after being consumed. Kafka retains the records according to its retention policy.

12. Kafka Is a Distributed Log

This is perhaps the most useful mental model for understanding Kafka.

Think of Kafka as:

A distributed, replicated, append-only log.

Not simply:

“A queue where messages disappear after consumption.”

For example

Kafka Log

0 → Event A
1 → Event B
2 → Event C
3 → Event D
4 → Event E

Multiple consumers can independently read this log.

Event A ─┐
Event B ─┤
Event C ─┼──► Consumer Group A
Event D ─┤
Event E ─┘
Event A ─┐
Event B ─┤
Event C ─┼──► Consumer Group B
Event D ─┤
Event E ─┘

The same event can therefore be processed by multiple independent consumer groups. This is one of Kafka’s biggest architectural advantages.

13. One Event, Multiple Consumers

Consider:

OrderCreated

The business may need:

Order Service
      │
      ▼
    Kafka
      │
      ├──► Payment Service
      │
      ├──► Inventory Service
      │
      ├──► Notification Service
      │
      └──► Analytics Service

Each service can consume the event independently.

The producer doesn’t need to know:

• who consumes the event
• how many consumers exist
• when they process it
• whether they process it immediately

This creates loose coupling.

However, Kafka does not magically remove coupling. The services are still coupled through the event contract. That means event schema design becomes an architectural concern.

14. Topic vs Queue

This is a common source of confusion.

A traditional queue often looks like:

Producer
   │
   ▼
 Queue
   │
   ▼
Consumer

Kafka supports a different model.

             Topic
          /    |    \
         ▼     ▼     ▼
        CG1   CG2   CG3

Each consumer group maintains its own position.

Therefore:

Payment Group
    └── processes OrderCreated
Analytics Group
    └── processes OrderCreated
Notification Group
    └── processes OrderCreated

The same event can participate in multiple independent processing pipelines.

15. A Simple End-to-End Kafka Flow

Let’s put the concepts together.

                Producer
                    │
                    │ Record
                    ▼
              ┌───────────┐
              │   Topic   │
              └─────┬─────┘
                    │
          ┌─────────┼─────────┐
          ▼         ▼         ▼
        P0         P1        P2
         │          │         │
         └──────────┼─────────┘
                    │
                    ▼
             Consumer Group
              /     |      \
             ▼      ▼       ▼
           C1       C2      C3

Internally, those partitions are distributed across brokers:

                  Kafka Cluster      
       Broker 1      Broker 2      Broker 3
      ─────────     ─────────     ─────────
       Topic P0      Topic P1      Topic P2
          │             │             │
          └─────────────┼─────────────┘
                        │
                        ▼
                   Consumers

This gives Kafka its combination of:

• distributed storage
• parallel processing
• fault tolerance
• scalability
• replayability

16. What Kafka Does NOT Guarantee Automatically

This is where architectural thinking becomes important.

Kafka does not automatically solve:

Exactly-once business processing

A consumer can process an event and crash before committing its offset.

The event may then be processed again.

Kafka
  │
  ▼
Consumer
  │
  ├── Process payment ✓
  │
  └── Crash before offset commit ✗
             │
             ▼
        Event processed again

Therefore:

Kafka reliability does not automatically mean business-level idempotency.

For critical operations such as payments, the application still needs an idempotency strategy.

17. Kafka Does Not Replace REST or gRPC

Another common architectural mistake is:

We are using Kafka, so we don’t need REST anymore.

Not necessarily.

Different communication models solve different problems.

REST

Good for:

Request → Response

gRPC

Good for:

Low-latency synchronous service-to-service communication

Kafka

Good for:

Event → Stream → Independent consumers

A real architecture may use all three.

Client
  │
  ▼
REST API
  │
  ▼
Order Service
  │
  ▼
Kafka
  ├──► Payment
  ├──► Inventory
  ├──► Notification
  └──► Analytics

Architecture is about choosing the right communication model — not choosing one technology for everything.

18. The Architect’s Mental Model

When looking at Kafka, don’t start by memorizing configuration properties.

Start with these questions:

1. What is my event?

Example: OrderCreated

2. What is my topic?

Example: order-events

3. What is my partitioning strategy?

Example: orderId

4. What ordering do I need?

Example: Events for the same order

5. How many partitions do I need?

Based on:

• throughput
• consumer parallelism
• expected growth
• ordering requirements

6. What happens if a broker fails?

This leads to replication.

7. What happens if a consumer fails?

This leads to consumer groups and offset management.

8. What happens if processing succeeds but offset commit fails?

This leads to duplicate processing and idempotency.

9. How long should events remain available?

This leads to retention design.

These are architecture questions — not Kafka syntax questions.

19. The Core Kafka Mental Model

If you remember only one diagram from this article, remember this:

                       PRODUCER
                           │
                           │ Record
                           ▼
                    ┌─────────────┐
                    │    TOPIC    │
                    └──────┬──────┘
                           │
              ┌────────────┼────────────┐
              ▼            ▼            ▼
          Partition 0  Partition 1  Partition 2
              │            │            │
              │            │            │
              ▼            ▼            ▼
          Ordered Log  Ordered Log  Ordered Log
              │            │            │
              └────────────┼────────────┘
                           │
                           ▼
                    CONSUMER GROUP
                     /     |      \
                    ▼      ▼       ▼
                   C1     C2       C3

And behind those partitions:

               Kafka Cluster  
     
         Broker 1   Broker 2   Broker 3
          │          │          │
          └──── Distributed ────┘
                 Storage

Once this mental model is clear, Kafka’s more advanced concepts become much easier.

20. What Comes Next?

In the next article, we’ll go deeper into the most important scalability mechanism in Kafka:

Topics → Partitions → Offsets

We’ll answer questions such as:

• How does Kafka distribute records across partitions?
• How does a partition actually provide ordering?
• How should you choose a partition key?
• Why can a bad partition key create a hot partition?
• How do partitions determine consumer parallelism?
• Why can’t one consumer group process a partition concurrently?
• How does partition count affect scalability?

Because in Kafka:

Partition design is application architecture.

Architect’s Takeaway

Kafka becomes much easier to understand when you stop thinking of it as a messaging queue and start thinking of it as a:

Distributed, replicated, append-only event log.

The fundamental building blocks are:

Broker
  ↓
Topic
  ↓
Partition
  ↓
Record + Offset
  ↓
Consumer Group

From these building blocks emerge Kafka’s major capabilities:

Scalability → Partitions

Ordering → Partition

Fault tolerance → Replication

Parallel processing → Consumer groups

Replay → Retained log + offsets

Loose coupling → Event-driven communication

And that is the foundation on which the rest of Kafka architecture is built.