Batch vs Streaming

Latency, cost and complexity trade-offs, and how to tell which one a pipeline actually needs

Data Platform Engineering · Part 6 of 40

A product manager asks for the collections dashboard to be "real-time." The team spends a quarter building a streaming pipeline, with a message broker, a stream processor, state stores and a new on-call rotation. Six months later, someone checks the usage logs. The collections team opens the dashboard once a morning, at 8:30, to plan the day's calls.

Meanwhile, the fraud check on new loan applications still runs as an hourly batch job. By the time it flags a burst of synthetic identities, eleven loans have already been funded.

Both teams picked the wrong processing model, in opposite directions. This article explains what batch and streaming really are, what each one costs, and how to match the model to the decision the data supports.

Where this fits

Part 2 introduced the lifecycle and listed frequency as the first decision in every ingestion pipeline. Part 3 showed why columnar files prefer large batched writes, and Part 4 showed how a database's write-ahead log records every change as it happens. Part 5 shaped data into facts and dimensions. This article answers the question we have been postponing: when does data move?

By the end of this article you should be able to:

• Define batch and streaming in terms of bounded and unbounded data, not tools.
• Place any pipeline on the latency spectrum, including the micro-batch middle.
• Explain why streaming costs more to build, run and debug.
• Work out the freshness a use case really needs, and pick the simplest model that meets it.

Kafka internals (Article 14) and stream processing techniques such as windows and watermarks (Article 15) get their own deep dives. Here we stay at the level of the decision.

Two ways to process data

Every dataset is either bounded or unbounded. A bounded dataset has an end: yesterday's applications, last month's repayments, a partner's nightly file. An unbounded dataset never ends: the stream of application events keeps coming for as long as the business exists.

Batch processing works on bounded data. It waits for a chunk of data to be complete, processes all of it in one run, writes the result, and stops. A scheduler starts the next run later.

Stream processing works on unbounded data. It runs continuously, handles each event (or a tiny group of events) as it arrives, keeps running state in memory, and never really finishes.

Batch collects data into bounded chunks and processes each chunk on a schedule. Streaming processes events continuously as they arrive.

Here is the idea that makes the rest of this article click: almost all data is born as a stream. Applications are submitted one at a time. Repayments arrive one at a time. The WAL from Part 4 records changes one at a time. Batch is not a different kind of data. It is a decision to wait, collect, and process later.

Batch is a stream you chose to wait for. The question is never "is my data streaming?" but "how long can I afford to wait?"

That framing also explains why the two models feel so different to build. Batch gets to see the whole chunk before it starts, so it can sort, join and aggregate freely. Streaming must decide what to do with each event before it knows what comes next.

The latency spectrum

Batch and streaming are not a binary choice. They are ends of a spectrum, and most real pipelines sit somewhere in between.

Processing models across the latency spectrum. Most analytical needs are met somewhere in the middle, not at the real-time end.

Four zones cover nearly everything you will build:

1. Daily or hourly batch. The classic scheduled job. Fine for reporting, finance, model training and anything reviewed by humans on a daily rhythm.
2. Micro-batch. Small batches every few minutes or seconds. Spark Structured Streaming's default mode works this way, as do frequent incremental loads into a warehouse.
3. Near-real-time streaming. Continuous processing with latency in seconds. Typical for operational dashboards, alerts and syncing data between systems.
4. Real-time streaming. Sub-second latency, usually in the request path of a product. Fraud checks, live pricing and instant notifications live here.

Latency is end to end

When someone asks "how fresh is this data?", the honest answer is the sum of every step: extraction, transport, processing, loading, and how often the consumer looks. A streaming pipeline that writes to a dashboard cached for an hour delivers hour-old data. A sub-second processor reading from a source that exports nightly is still a nightly pipeline.

Measure freshness where the data is used, not where it is processed. Article 20 turns this into a monitored metric.

How batch works

A batch job takes a bounded slice of data, usually defined by time, and processes it from start to finish. The slice is often a date partition: "process everything submitted on 27 October."

-- Daily batch: rebuild one day of the application fact table
-- :run_date is supplied by the scheduler (e.g. 2026-10-27)
DELETE FROM fct_loan_applications
WHERE submitted_date = :run_date;

INSERT INTO fct_loan_applications
SELECT ...
FROM stg_applications
WHERE submitted_date = :run_date;

Deleting and rewriting the whole partition makes the job safe to run twice, which is the property that makes batch so forgiving. If the logic was wrong, fix it and rerun the affected days. If a source was late, rerun yesterday. Article 22 covers idempotency and backfills in depth.

Batch has real strengths:

• Simple mental model. Input is complete and fixed before the job starts.
• Efficient. Processing a big chunk at once suits columnar files and parallel engines (Parts 3 and 4).
• Easy to test and debug. You can rerun the exact same input and inspect every step.
• Cheap to operate. Compute runs for minutes, then shuts off. Failures can usually wait until morning.

And real weaknesses:

• Latency is built in. Data is at least one schedule interval old, plus run time.
• Load spikes. Everything runs at once, often at the same hour as everyone else's jobs.
• Boundary problems. An event submitted at 23:59:58 that arrives at 00:00:03 can land in the wrong day, or no day at all.

How streaming works

A streaming pipeline has three parts: a durable event log that holds incoming events (Kafka or Kinesis, covered in Article 14), a stream processor that runs continuously, and one or more sinks where results land.

Get Syed Noman Ahmed’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

What makes streaming hard is not speed. It is that the processor has to make decisions about incomplete data. Four concepts sit at the heart of every streaming system.

Event time vs processing time

Event time is when something happened: the moment the customer pressed "submit." Processing time is when the pipeline saw it. In a perfect world they would match. In practice, mobile apps go offline, networks retry and partitions lag, so events arrive late and out of order.

Events rarely arrive in the order they happened. Counting by processing time puts the late application in the wrong window.

If you count "applications per minute" by processing time, a phone that reconnects after a tunnel shifts its applications into the wrong minute. Counting by event time is correct, but then the processor must decide how long to wait for stragglers. That decision, made with windows and watermarks, is the subject of Article 15.

State

Many useful questions need memory: "how many applications from this device in the last ten minutes?" The processor must keep that count somewhere, survive crashes without losing it, and rebuild it after a restart. State is what turns a simple streaming job into a distributed database you now operate.

Delivery guarantees

When a component fails and retries, an event can be processed zero times, once, or more than once. Systems advertise at-most-once, at-least-once or exactly-once behavior. Exactly-once is achievable inside a well-configured stream processor, but the guarantee usually ends at the sink. Writing to an external system twice still writes twice unless the write itself is idempotent.

It never stops

A batch job that fails at 2 a.m. can wait. A streaming job that fails at 2 a.m. builds a backlog every second it is down, and the backlog has to be drained when it comes back. Streaming pipelines need monitoring for consumer lag, alerting, and someone on call.

Same logic, different trigger

Modern engines narrow the gap in code. In Spark, the same transformation can run as a batch job or a stream by changing how it reads and when it fires:

# Batch: read what exists now, process, stop
apps = spark.read.table("raw_applications")

# Streaming: read new data continuously
apps = spark.readStream.table("raw_applications")

summary = apps.groupBy("channel").count()

(summary.writeStream
    .outputMode("complete")
    .trigger(processingTime="5 minutes")  # micro-batch every 5 minutes
    .toTable("channel_counts"))

The code looks almost identical. The operational reality behind it, with checkpoints, state, lag and on-call, is not.

The real trade-offs

Teams usually compare batch and streaming on latency alone. Latency is the one dimension where streaming always wins, which is exactly why the comparison needs the other rows.

Batch and streaming compared beyond latency. Micro-batch sits in between on almost every row.

Cost

Streaming costs more in three ways that compound:

• Always-on compute. A batch job runs for 20 minutes a day. A stream processor runs for 1,440 minutes a day, sized for peak traffic, even at 4 a.m.
• Storage overhead. Frequent small writes create the small-files problem from Part 3, so streaming pipelines need compaction jobs, which are batch jobs.
• People. Streaming needs specialized skills and an on-call rotation. That is usually the largest cost and the one least often budgeted.

A rough rule of thumb: each step down the latency spectrum, from daily to hourly to minutes to seconds, raises cost and complexity by more than the previous step did.

Correctness and reprocessing

In batch, fixing a bug means rerunning a partition. In streaming, fixing a bug means replaying the event log from an earlier point, rebuilding state, and reconciling results already sent to downstream systems. That only works if the log retained the events long enough, which is one more reason to keep the raw data (Part 2).

Testing and debugging

A batch job can be tested against a fixed input file. A streaming job has to be tested against timing: late events, duplicates, out-of-order arrivals, restarts mid-window. Bugs that only appear under specific arrival orders are notoriously hard to reproduce.

Micro-batch: the pragmatic middle

Micro-batch processes small bounded chunks on a short, repeating schedule: every minute, every five minutes, every fifteen. Each run is a normal batch job over the data that arrived since the last one.

It inherits most of batch's strengths, such as simple logic, easy reruns and efficient writes, while bringing freshness down to minutes. It also inherits some of streaming's concerns. Runs must track exactly where the last one stopped, and state that spans runs has to be stored somewhere.

For most analytical use cases, a five-to-fifteen-minute micro-batch is the sweet spot. Few humans act on dashboard changes faster than that, and the cost difference compared with true streaming is large.

Warehouses increasingly offer micro-batch as a declarative feature. Snowflake's dynamic tables, for example, let you set a target lag and leave the refresh schedule to Snowflake, processing only what changed where it can. You state how stale the result may be; the platform decides how often to run.

Choosing: start from the decision

The most useful question is not "batch or streaming?" It is "what decision does this data drive, and how quickly does acting on it create value?" Freshness has value only if someone, or something, acts faster because of it.

Lending use cases mapped to the freshness they actually need. Only decisions made inside the customer journey need true streaming.

Notice the pattern in the table. Machines acting in the request path need streaming. Humans reviewing data almost never do. The collections team from the opening plans their day once each morning, so a daily batch ready by 8 a.m. serves them perfectly. The fraud check runs while the applicant is waiting on a spinner, so an hour is far too long.

A simple decision guide. Work down the questions and stop at the first model that meets the need.

Questions to ask before saying "real-time"

When a stakeholder asks for real-time data, these questions usually reveal the real requirement:

• What action will you take when the number changes?
• Who or what takes it: a person, a scheduled process, or an automated system?
• How quickly is that action taken today?
• What does a delay cost? A worse customer experience, money lost, a missed regulatory deadline?
• Is the source itself real-time? If the servicing system exports once a night, no pipeline can make its data fresher.

Default to the slowest model that meets the need. You can always speed a pipeline up; slowing down a streaming estate is much harder.

Most platforms run both

In practice, the choice is made per pipeline, not per platform. A mature lending platform streams application events for fraud scoring and customer notifications, runs micro-batches for intra-day operational views, and runs daily batches for finance, regulatory reporting and model training.

Often the same events feed both paths. Application events flow through the event log to a stream processor for instant decisions, and are also landed in the lake, compacted into Parquet and processed in batch for analytics.

One event stream, two paths. The streaming path serves decisions in seconds; the batch path serves analytics and reporting from the same raw data.

Running two paths raises an obvious risk: two implementations of the same logic that slowly drift apart, producing two different fraud counts. How architectures handle this, including the Lambda and Kappa patterns, is covered in Article 25. Writing streams straight into open table formats, which blurs the line further, comes in Article 10.

What is changing

The engineering gap between batch and streaming is shrinking. The decision framework in this article is not.

• Spark has a real-time mode. Apache Spark 4.1, released in December 2025, added Real-Time Mode to Structured Streaming. It runs the same APIs continuously instead of in micro-batches, and Databricks announced its general availability on its platform in March 2026. The pitch is one engine for both ends of the spectrum, instead of Spark for batch and Flink for low latency.
• Warehouses are making incremental processing declarative. Snowflake’s dynamic tables, and similar declarative pipeline features elsewhere, let you specify a freshness target instead of hand-building schedules.
• Kafka keeps broadening its scope. Apache Kafka 4.2, released in February 2026, made share groups (queues for Kafka) production-ready, so queue-style work and event streaming can share one system. More on that in Article 14.

These changes lower the cost of building streaming pipelines. They do not lower the cost of running them, debugging them or explaining their numbers. Pick the latency the business needs first, then let the tooling make it cheaper.

Common mistakes

• Treating “real-time” as a feature request. It is a cost decision. Ask what action the freshness enables before building anything.
• Streaming data that humans read once a day. You pay for continuous compute and on-call to serve a morning meeting.
• Hourly batch in the customer journey. The opposite failure: decisions that need seconds get data that is an hour old.
• Ignoring event time. Counting by arrival time puts late events in the wrong window and makes numbers shift between runs.
• Assuming exactly-once end to end. Most guarantees stop at the sink unless writes are idempotent.
• Forgetting the batch half of streaming. Compaction, backfills and reconciliation are batch jobs every streaming platform still needs.

What comes next

We now know when data moves. The last article in Part 1 looks at the order of operations: ETL vs ELT. Article 7 explains why data used to be transformed before it was loaded, how cloud warehouses flipped that order, and what changed for data teams as a result.

Key takeaways

• Batch processes bounded data on a schedule; streaming processes unbounded data continuously. Almost all data is born as a stream, and batch is a choice to wait.
• Latency is a spectrum, from daily batch through micro-batch to sub-second streaming. Measure it end to end, where data is used.
• Streaming costs more than its compute bill: state, late and out-of-order events, harder reprocessing, harder testing and an on-call rotation.
• Micro-batch is the pragmatic default for most analytical needs that are “fresher than daily.”
• Start from the decision. Machines acting in the customer journey need streaming; humans reviewing data almost never do.
• Most platforms run both, often from the same events. Default to the slowest model that meets the need.

This is Part 6 of the Data Platform Engineering series. Previously: Part 5, Data modeling fundamentals. Next up: Part 7, ETL vs ELT.