Amazon Kinesis: Real-Time Streaming on AWS

DataForge AWS Data Engineering Series - Blog 8 of 20

The consumer had been running fine for months. Then, over the course of one Tuesday afternoon, records started arriving late — first by seconds, then by minutes, then by hours. Nobody had changed the consumer code. Nobody had changed the producer. The stream’s IteratorAgeMilliseconds metric, if anyone had been watching it, would have shown the problem building for hours before it became visible downstream.

The actual cause: a marketing campaign had doubled event volume overnight, pushing the stream past the throughput a single shard could handle. The shard didn’t reject the extra records — it accepted them, but the consumer, bound by that same shard’s read throughput limit, fell further behind with every batch. By the time someone noticed the dashboard was showing stale data, the consumer was four hours behind real time, and catching up took most of the next day.

This is the defining characteristic of Kinesis failures: shard capacity limits don’t produce errors when exceeded on the write side — they produce growing lag on the read side, invisible unless you’re specifically watching for it, and the fix (adding shards) doesn’t happen automatically unless you’ve configured it to.

Let’s go deep.

TL;DR

Kinesis Data Streams is AWS’s native real-time streaming service, built around shards — fixed units of throughput capacity…