Layer 2: The App Gets Popular

Layer 2 of Down the Stack: Kafka in Nine Layers. One partition, one consumer, and a growth chart that is pointing in the wrong direction.

Layer 1 left us with a working system: one topic, one partition, one consumer, counts landing in Mysql. At current traffic this is fine. But the product is growing, and it is worth thinking through what “traffic grows” actually means for this architecture before it happens, not after.

Here is the scenario to plan for: the app gets featured, traffic spikes 40x in a couple of hours. The producer side is not a concern, appending to a log is cheap and the broker barely notices. The consumer side is the problem. It processes events one at a time, and with an order-of-magnitude jump in ingest rate, the lag between “event happened” and “event counted” starts growing. The instinctive fix is to run more consumers, and here the one-partition decision comes due: within a consumer group, a partition is read by exactly one consumer at a time. A second page-counter instance connects, gets assigned no work, and sits idle. The unit of consumer parallelism in Kafka is the partition.

Understanding why unlocks most of Kafka’s scaling model.

Partitions: the unit of everything

A topic in Kafka isn’t really one log. It’s N logs, called partitions, and each partition is the thing that’s actually ordered, actually stored on disk, and actually assigned to consumers.

Our topic has one partition, so it behaves like one log. Within a consumer group, a partition is read by exactly one consumer at a time. One partition, one reader; the second instance has nothing to claim.

The unit of consumer parallelism in Kafka is the partition.

So we need more partitions. How many?

“More than one” is not a plan. Partitions have downstream consequences: every partition is files on disk, replicas across brokers, and (critically for this series) a state store shard per Streams application. Overshooting now means carrying unused overhead forever; undershooting means redoing this work later, which breaks key routing in ways Layer 5 will make concrete.

Back-of-envelope: Assuming the app is grows toward 10,000 events per second at peak. Benchmarking the consumer under sustained load, with the MySQL write and JSON deserialization in the critical path, shows roughly 1,000 events per second per consumer thread. That gives a raw minimum of 10. The benchmark number is optimistic: real traffic bursts rather than sustains, and production adds garbage collector pauses and connection pool contention that a controlled run never sees. Round up to 12 because it divides evenly by 3, 4, 6 (clean assignments for our likely consumer counts) and in a three-broker cluster places exactly four leader partitions per broker.

One thing that number commits us to: partition count should stay constant for as long as this topic is part of a Streams topology. Partitions are how Kafka distributes work, and state stores are sharded in lockstep with partitions. Change the partition count after the fact and hash(key) % N routes events to different partitions than the state that was built for those keys. This produces silent misses in every join and aggregation downstream. The right time to think about partition count is before the first record is written.

The commitment doesn’t stop at the ingestion team’s own consumers. Kafka Streams requires co-partitioning for stateful joins: any two topics joined in a topology must have the same partition count and the same partitioner, so that records with the same key land on the same task. Twelve partitions on events means every downstream Streams topology joining against it needs its partner topics to also have twelve partitions. A team that wants to join events with a six-partition experiment-assignments topic has to resolve the mismatch: either align the topics upfront, or rekey one stream inside the topology (which causes Streams to create an internal repartition topic automatically, adding a write step per record). Teams that only consume without joining are unaffected beyond the parallelism ceiling of twelve consumers. Teams using a GlobalKTable (which broadcasts the full table to every task and waives the co-partitioning requirement) are also clear. Everyone else inherits the number chosen today.

kafka-topics.sh --bootstrap-server localhost:9092 \
  --create --topic events --partitions 12

Twelve partitions means up to twelve consumers in a group, each owning a slice. Run four instances of the page-counter and each gets three partitions; the lag drains in minutes.

4 different jvms/threads of the page-counter service, each reading from a different paritition

But we’ve just changed something subtle and important. With one partition, every event was in one ordered sequence. With twelve, Kafka guarantees order within a partition and says nothing about order across partitions. An event in partition 3 and an event in partition 9 have no defined relative order, no matter when they were produced.

Which raises the question Layer 1 ended on: when an event arrives, which partition does it go to?

The key decides

Every Kafka record has an optional key. When routing a record to a partition, the partitioner uses that key. Three policies are worth knowing:

• No key (sticky/round-robin): Records are distributed across partitions for even throughput. A batch sticks to one partition briefly (for batching efficiency), then moves to the next. No ordering guarantees between any two records.
• Key-based (murmur2 hash): partition = murmur2(key) % partition_count. Records with the same key always land on the same partition. This is Kafka's per-key ordering guarantee: every event for a given key arrives in the order it was produced, on one partition, processed by one consumer. It is also the foundation of the co-partitioning requirement in joins, which Layer 5 will explore in detail.
• Custom partitioner: Implement the Partitioner interface for business-aware routing (dedicated partitions for high-priority tenants, for example). Custom partitioners are the sharpest edge in the toolbox: you now own the contract of what "same key implies same partition" means, and getting it wrong is silent.

For our analytics events, key-based partitioning seems right. The ordering guarantee is what we need. Did the user view the pricing page before or after the docs page? Events scattered keylessly across partitions will be processed out of order by different consumers.

The remaining question is which key to use.

The obvious candidate is user_id, and we can't use it: half our traffic is anonymous. Visitors browse logged-out, and user_id doesn't exist until they sign in. What every event does have, from the first page load, is the session: the tracking snippet mints a session_id cookie on arrival and sends it with every event.

kafka.send("events", event.sessionId(), json.writeValueAsString(event));

One argument added to the send() call. Every session's events now arrive in order, on one partition, and funnels work for anonymous and logged-in visitors alike. Sessions are the unit of analysis in this product, so the sessionId is the key.

A brief side note, because someone always asks: if ordering is good, why not order everything? Because a topic with global ordering is a topic with one partition, which is a system with one consumer, which is the lag problem we just escaped. Global ordering and parallelism are the same dial turned in opposite directions. The skill isn’t maximizing ordering; it’s knowing the smallest unit that genuinely needs it.

Consumer groups, properly this time

Layer 1 hand-waved group.id as a named bookmark. Here's the fuller picture: a consumer group is a set of consumers sharing one bookmark and one workload. Kafka divides the topic's partitions among the group's members, and when membership changes (an instance deploys, crashes, or scales), it redistributes them. That redistribution is called a rebalance. For now, think of it as brief musical chairs; several later layers will give it the respect it deserves.

The part that makes groups quietly brilliant is what happens between groups: nothing. Each group has its own offsets, its own pace, its own members. The same data, read independently by everyone who needs it.

Get Karan Patel’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

That matters because the page-counter no longer has the topic to itself. Success attracted teams:

• The dashboard team wants raw events for a live view.
• The growth team wants to alert when signups dip.
• The data team wants everything, eventually, in the warehouse.

In a queue world, each of these is a request to the ingestion team: build us a feed, fork the stream, add a binding. In Kafka, each team picks a group.id and starts reading. The prologue called the topic an organizational contract; this is what that looks like in practice.

// Each team writes their own listener against the same topic
@KafkaListener(topics = "events", groupId = "page-counter")
public void count(ConsumerRecord<String, String> r) { /* Layer 1's counter */ }
@KafkaListener(topics = "events", groupId = "live-dashboard")
public void display(ConsumerRecord<String, String> r) { /* real-time feed */ }
@KafkaListener(topics = "events", groupId = "signup-alerts")
public void alert(ConsumerRecord<String, String> r) { /* only cares about SIGNUP events */ }

The broker tracks three independent offsets. signup-alerts only reads signup events, so it falls behind the others in offset terms and nobody cares: Kafka doesn't know which events you'll process inside the listener, only that you read the record:

group "page-counter"   -> offset 84,213,991
group "live-dashboard" -> offset 84,214,002
group "signup-alerts"  -> offset 84,002,117

Consumer threading: what you are actually running

A consumer group is one of the most important concepts to understand regards to kafka and its threading model is where some of the subtler failures later in this series are born.

Each consumer is a poll loop: a thread (or process, or JVM) that calls poll() on a schedule, receives a batch of records, processes them, and calls poll() again. The broker doesn't push; the consumer pulls.

Two separate mechanisms tell the broker a consumer is alive, and they catch different failure modes:

• Heartbeat thread: a background thread sends periodic heartbeats to the broker independently of the poll loop. If the broker stops receiving them for longer than session.timeout.ms (default 45 seconds), it declares the consumer dead and triggers a rebalance. This catches hard failures: process crashes, GC freezes long enough to stall the heartbeat thread, network partitions.
• Poll interval: the broker also tracks how long it has been since the last poll() call. If that gap exceeds max.poll.interval.ms (default five minutes), the consumer is evicted even if heartbeats are still arriving. This catches a different problem: a consumer that is alive but stuck processing a batch so large or slow that it never comes back for more records.

The distinction matters because the fix is different. A session timeout means the consumer process is unhealthy. A poll interval breach means the processing inside the listener is too slow for the batch size, and the right lever is max.poll.records or making the processing faster, not the timeout.

Consumer instances in the same group can be different JVMs on different machines, for e.g. all page-counter replicas share a group.id and Kafka treats them as a pool of active poll loops. They can also be threads within the same JVM: spring-kafka's @KafkaListener on a single-threaded container is one poll loop, one thread, one consumer. If that consumer owns three partitions, their records are processed sequentially on that thread, partition 3's batch first, then 7's, then 11's, one after another. For concurrency inside one JVM, ConcurrentKafkaListenerContainerFactory.setConcurrency(N) creates N separate consumer instances, each with its own poll loop and assignment.

The poll interval check is at the consumer level, not the partition level.

Processing one partition’s batch can’t stall without affecting the others on the same consumer. If a slow database write or an external API call inside the listener takes longer than max.poll.interval.ms, the broker evicts the entire consumer, not just the slow partition, the whole consumer and all its healthy partitions. The rebalance that follows pauses work across the group until it resettles. Slow downstream calls inside a listener are not a latency problem; they are a group-health problem. This pattern has further implications in Layer 7 when we add gRPC calls inside the processing loop.

The config that governs this, worth setting explicitly rather than leaving at the default:

spring:
  kafka:
    consumer:
      max-poll-interval-ms: 30000    # default is 300000 (5 min); size to your actual processing time
      max-poll-records: 100          # batch size per poll; smaller batches = more frequent polls

Kafka Streams manages threading similarly: num.stream.threads (default one) controls how many internal threads the application runs, each owning a subset of assigned partitions, under the same poll-interval contract.

Sidebar: when to not use kafka

Around this time in our story, a neighboring team sees the shiny new event pipeline and asks to use it for webhook delivery: customers register URLs, the product POSTs events to them. Retry with backoff if an endpoint is down, rate-limit per customer, give up after 24 hours, let customers replay missed deliveries. “It’s events going out instead of in. Just another topic, right?”

It’s a reasonable instinct, and it leads somewhere bad, and walking through why is one of the better ways to understand what Kafka is.

A partition is an ordered lane that must be consumed in order. Webhook delivery is the opposite of that: thousands of independent jobs, each with its own retry clock, no meaningful order among them. Put them in a partition and one customer’s dead endpoint becomes everyone’s problem: the consumer hits the failing delivery, and now it either blocks the whole lane retrying (head-of-line blocking: every webhook behind it waits), or it skips ahead and has to track out-of-band which offsets still need work, at which point you’ve built a job queue on top of Kafka, badly, with the log fighting you the whole way. Per-message retry schedules, per-customer rate limits, priorities, delayed delivery: none of these map onto “read a file forward.”

A boring job queue (SQS, RabbitMQ, even a database table with a worker pool) handles all of it natively, because per-message lifecycle is exactly what those tools model and exactly what Kafka deliberately gave up to get its throughput.

The lesson generalizes: Kafka is a log, not a job queue. Logs are for facts that many readers consume in order; job queues are for tasks that workers complete independently. Most “Kafka was a mistake here” stories start with confusing the two. The webhook team got a job queue and everyone stayed friends.

Where this leaves us

The pipeline scales now. Twelve partitions, four counter instances, per-session ordering for the funnel team, and three consumer groups reading independently. The system survived its first scaling problems.

The next problem doesn’t announce itself with lag graphs. It arrives quietly, in a Slack thread, two weeks after the mobile team ships SDK v2.

The dashboard team’s numbers and the growth team’s numbers have drifted apart. Same topic, same events, different answers. After a day of confused digging, someone finds it: v2 renamed the url field to page_url. New-SDK events have been flowing in with a shape one consumer handles and another doesn't, and nothing anywhere considered that an error. Every team parses the JSON its own way, every parser fails its own way, and the topic, faithful append-only log that it is, accepted all of it without comment.

Three teams, one topic, zero contracts. Layer 3 is about fixing that, and about the day we learned what a single malformed message does to a partition.

Read the full code for this layer at : https://github.com/Panda0Monium/kafka-layer-by-layer/tree/master/layer2

Down the Stack: Kafka in Nine Layers · Prev: Layer 1 · Next: Layer 3, Clients Send Garbage