Batch-Processing

So washing the dishes could teach us software engineering?

“Okay, this wash-as-we-use dishwashing stuff I have been using since Wednesday, has been avoiding me the stress of having to wash so many plates at night, but my goodness! This Kitchen is starting to become my second room, and making it hard to focus on other things during the day.”

I tell myself while staring at the bubbles foaming up on a plate under a running faucet, standing in front of the sink, with a foamed sponge on my right hand, a plate on my left hand and pile of other plates on the sink.

As a child, You see I could do any chore, and by any, I mean anyy… but washing the dishes? my goodness, it has always been a challenge to me.
So I came up with many tactics to avert this as a kid, but it never really worked, especially during the holidays when the house would be filled and we were 6 children so ever meal meant I had to wash an average of 18+ plates a day and as someone who dreads dishwashing as much as I do, man I had to try escape it.

• One time, I tried paying off my immediate younger Brother, we were 5 kids and he was the last of us.
  I was eight at the time and we agreed he took the meat in 3 of my meals for any-time he helped me wash plates.
  Man, this went on for a while and life was good till my African parents found out and called-off our agreements.
  According to them every child has a stage in their life when they are meant to do the dishes and I won’t escape mine, my goodness! 😭
• More time went by, and on a random Tuesday, after hearing my friend Daniel tell me about how, He also detested dish-washing and how his elder sister gave him the advice of wash-as-we-use.
  The base idea was to monitor how frequently people used dishes and make sure I cleared the sinks immediately someone used up a plate. This helped me avoid having to wash a large amount of dishes at once, and surprisingly made dish-washing a lot more easier and sometimes fun.

The Twist

You see a few days went by, and I realized this technique was making me spend a whole lot of time in the kitchen, my siblings were fund of drinking cereal mid-day or using something that required them using a lot of plates. And slowly I found this exhausting.
I finally understood why it was set standard to wash plates at set time, like at evening or at the end of the day.

The Turn

You see this pattern of waiting for dishes to accumulate before washing them extends beyond my childhood kitchen to Backend Systems involving procesing along large amounts of growing data (high-throughput event data), and here’s how it works at this scale. Building a product like Kronos Change-collection system, for instance.

What is the Kronos Change-Collection System?

It is responsible for the secure storing of developers git commit activities and making sure it gets to the analytical Engine of Kronos — which tracks user commit activity along selected repositories that users explicitly opt into tracking called Krons, which collects activity metrics via webhooks & scheduled tasks, and generates actionable insights for developers to work more efficiently from their accumulated data.

To do this the System, temporarily stores all sanitized commit webhook payload information in redis under a unique key for each user
kron:{userId}:commits as a JSON list, and then periodically adds registers this data in a Job queue to be processed by workers.

diagram illustrating the System flow of the Kronos Change Collection System

Think of redis as a data-store which arranges data in key-value pair, a key serves as an identifier for a value.

A Job-Queue is a group of operations being processed by a worker after a given time interval, like waiting till evening to wash the dishes instead of interrupting every of your daily activity because someone used a new plate, and you could liken yourself to be a worker.

This sequence of storing high throughput (growing data) in a data-store, and then processing it at intervals by using a Queue and a Worker is called Batch Processing.

Other Applications of Batch Processing

Beyond Kronos, Batch Processing is employed by other industries like Spotify:
The platform serving 700M+ users and processing trillions of events every year, leans heavily on batch processing too to generate wrapped‑style insights, recommendations, and product analytics. Let’s walk through the exact technology stack.
Source: As of mid 2017, spotify ran 700+ production pipelines with their largest batch job using 800 nl-highmen-32 workers

The Big Picture: Batch vs Real‑Time

Spotify doesn’t react to every single play, skip, or save in real time.

Instead:

• Event ingestion: user actions (play, track‑change, like, save) are logged as events and stored in queues or distributed messaging systems.
• Batch jobs: once per hour or per day, these events are read back in large, bounded chunks and processed together.

This is similar to:

• You letting plates accumulate in the sink all day, then doing a single, long dish‑washing session in the evening.

Reacting to every single plate in real time would be like a real‑time / streaming pipeline: you run, grab the sponge, and wash the moment someone puts a plate down.

Spotify mixes both, but for analytics, recommendations, and reports, batch is the go‑to.

The Core Technical Layer: Scio + Apache Beam

At the heart of Spotify’s batch processing is Scio, a Scala API built on Apache Beam.

• Apache Beam is the unified programming model for both batch and streaming. Engineers write pipelines that can run either as:
• A bounded pipeline (a finite set of data, e.g., “all events from yesterday”), or
• An unbounded pipeline (a continuous stream, e.g., “all future events from this Kafka topic”).
• Scio is Spotify’s own Scala wrapper on top of Beam. It gives engineers type‑safe, functional‑style combinators (map, flatMap, groupBy, join, etc.) to transform events, counts, and aggregates in purely functional code.

This means:

• A data engineer writes one pipeline once in Scio.
• Spotify can then run that same pipeline as:
• A batch job (e.g., “calculate top artists per user for last week”), or
• A streaming job (e.g., “update Now‑Playing stats in real time”).

Conceptually, this is like deciding whether you want to:

• Run a batch wash (all accumulated plates at once),
• Or a stream‑style wash (plate‑by‑plate as they’re used).

Execution Engines: Dataflow and Flink

Writing in Scio is only half the story. You still need a runtime to actually execute the pipeline.

Get David Chukwuemeka’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Spotify mainly runs their batch (and streaming) jobs on:

• Google Cloud Dataflow
• Apache Flink

1. Google Cloud Dataflow

• Dataflow is a fully managed service on Google Cloud that automatically spins up and scales a cluster of workers based on the size of your data.
• A typical batch job might scan hundreds of billions of rows (user activity logs) and aggregate them into per‑user or per‑session stats.
• Spotify has used configurations like 800+ high‑memory workers on GCP for large recommendation‑oriented batch jobs.

From our perspective, Dataflow is like:

• Hiring a dishwasher army for one night,
• They grab all the accumulated plates, organize them onto multiple racks, and run them through the dishwasher in parallel,
• You don’t worry about how many machines they use; you just get the job done.

2. Apache Flink

• Flink is another distributed data‑processing engine that can run both bounded (batch‑like) and unbounded (streaming) jobs.
• Some Spotify use cases run Scio‑backed pipelines on Flink, especially when they want to reuse the same logic across batch and streaming while keeping strong consistency and fault‑tolerance guarantees.

Think of Flink as:

• A more flexible, programmable dishwasher that can either:
• Run a batch wash (load everything, then spin), or
• Run a continuous wash (plates go in, racks revolve, plates come out clean).

Storage and Data Warehouses: BigQuery &Bigtable

Batch processing is only as powerful as the data it can read from and write to. Spotify’s stack ties nicely into:

1. BigQuery (Google Cloud Data Warehouse)

• Many of Spotify’s analytics and reporting pipelines write their results into BigQuery, a serverless, petabyte‑scale data warehouse.
• High‑level metrics like “top tracks of the week,” “listen‑time per user,” or “Podcast engagement stats” are often stored here and then queried by dashboards or recommendation services.

Think of this like:

• After you’ve washed all the plates, not just putting them away randomly,
• But putting them into well‑organized shelves where anyone can easily find the “special plates” or “holiday plates” without digging through the whole kitchen.

2. Bigtable and Other Databases

• Event‑level data (thousands or millions of user actions per second) are often stored in Bigtable or similar distributed databases.
• Scio / Beam pipelines will scan these tables in batch mode, group events by user, session, or window, and push aggregates back into BigQuery or feature stores.

In kitchen terms:

• Bigtable is your deep sink with a pile of dirty plates.
• Your Scio pipeline is the dish‑washing routine: fetch plates, scrub, rinse, stack them neatly in BigQuery.

How Events Move Through the System

Here’s a realistic flow of how Spotify’s batch processing pipeline looks from ingestion to analytics:

1. Event ingestion

• Every time a user plays a track, skips, or saves, Spotify logs an event.
• Events go into a pub‑sub / Kafka‑like system (e.g., Google Cloud Pub/Sub), a queue that holds events until they’re read.

2. Batch windowing

• A batch job is scheduled to run every hour or every 24 hours.
• It reads events for a bounded window (e.g., “all events from 00:00–23:59 UTC on May 8, 2026”).

3. Processing in Scio (Beam)

• The pipeline:
• Deserializes events,
• Groups them by user and time window,
• Aggregates streams, likes, skips, saves, and durations.
• Code might look like:

events
  .groupBy(_.userId)
  .mapValues(_.sortBy(_.timestamp))
  .flatTransform(analyzeSession)
  .saveAsBigQuery("spotify.user_stats_daily")

This is illustrative, but aims to closely mirror actual Scio patterns.

4. Write to BigQuery

• The output of the job is written to BigQuery tables that product teams and recommendation services query.

5. Use in recommendations and “Wrapped”

• High‑level metrics (listen‑time, top‑artists, most‑replayed‑songs) are computed from these batched tables.
• These tables feed:
• Personalized recommendations,
• “Discover Weekly,”
• “Wrapped”‑ style year‑end reports.

Tying Back to Kronos

If we project Kronos‑style change‑collection system onto Spotify’s stack:

• Redis list kron:{userId}:commits is like a pub/Sub / Kafka topic holding user events.
• Node-Cron Periodically pushing that data into a BullMQ job queue is like a Scio/Beam pipeline scheduling that reads events over a bounded window.
• BullMQ Workers processing the queue is like Dataflow/Flink workers executing the batch job.
• Writing to Kronos analytical Engine is like Spotify writing to BigQuery, where aggregates live for dashboards and models.

Both are using bounded batch processing:

• Krons: waits for all developer commit data to accumulate, then analyzes them.
• Spotify: waits for all listening events in a time window, then aggregate them once.

Why This Matters

Batch processing lets organizations:

• Keep costs and resource usage predictable: you pay for a few big, scheduled jobs, not a ton of tiny real‑time calls.
• Provide high‑quality analytics and recommendations that are strongly consistent because they’re computed over large, well‑defined windows.

So next time you catch yourself:

• “Ah, I’ll just leave the dishes; I’ll do them later…”
• Remember: you’re not procrastinating, you’re just implementing a bounded batch job on your kitchen sink.

And Kronos, Spotify and a couple other Software Engineering backend teams is doing the exact same thing, just at a larger scale.