How a Deadlock Froze Blinkit’s Supply Chain
A silent deadlock in our query engine was stalling inventory replenishment jobs with no error, no crash — just infinite waiting. This is the story of how we found it, traced it to an open-source bug, and fixed it upstream.
TL;DR
Trino’s Hudi connector used a single thread pool for both producing file splits and signalling when there was room for more. Under load, every thread ended up waiting for a signal that had no thread left to run it. The fix was to switch the producer side to a cooperative scheduling pattern: yield the thread when the buffer is full, and resume when space opens.
Our inventory replenishment pipeline was frozen. CPU was idle. Memory was fine. There were no errors anywhere. Queries just… stopped moving.
The first signal was a long queue on one of our analytics clusters. Queries were piling up. Inventory replenishment jobs — the jobs that decide how much stock every warehouse and store needs to hold — were delayed. Blinkit’s supply chain was being impacted. Dashboards were turning amber, but nothing was crashing. That was the unsettling part.
Investigation: Resources Doing Nothing
The affected cluster runs analytical workloads on Trino, reading data stored in Apache Hudi tables on S3. The natural first instinct in a queue build-up is to look at resource saturation — a CPU spike, memory pressure, network bottleneck. There was none of that. The cluster was sitting largely idle, with CPU barely above baseline and heap usage well within limits.
Every new query touching a Hudi table joined the queue and stayed there indefinitely. Queries that were already mid-execution completed fine. Only freshly submitted ones were affected. And crucially, there were no errors. No timeouts, no exceptions in the logs — just silence and a growing backlog.
A thread dump — a snapshot of what every thread in the process is doing right now — was our next move. It showed dozens of producer threads all stuck in the same parked state, waiting on the exact same internal signal. Not running. Not erroring. Just waiting, indefinitely, for something that was never going to arrive.
Architecture
At this point, the thread dump had given us a symptom, but not the shape of the system around it. To understand why a waiting thread could freeze planning completely, we need one small piece of Trino/Hudi architecture.
When you run a SQL query against a Hudi table, Trino doesn’t execute it in one shot. First it needs a plan: figure out which physical data files on S3 are relevant to the query. A large table might be split across hundreds of date or region partitions, each containing many Parquet files. Trino calls each such file chunk a split — it’s the unit of work handed to a worker node.
This file-listing step — split enumeration — runs asynchronously in the background while the query is being set up. A pool of producer threads lists partitions and pushes file references into a fixed-size queue. For the rest of this story, call that queue Split Buffer. On the other side, the query engine pulls from Split Buffer — these are the consumers. Split Buffer exists to stop planning from flooding workers with too much work at once. Crucially, both the producers and the consumers share the same fixed pool of worker threads.
Diagnosis: The Deadlock Circle
The thread dump made the picture clear. Producer threads were filling Split Buffer and then blocking — holding their thread, doing nothing — waiting for Split Buffer to drain. Split Buffer could only drain if the consumer side got a thread to run. But every thread in the pool was already held by a blocked producer. A perfect circle.
The fatal design flaw was deceptively simple: producers and consumers shared the exact same thread pool. When Split Buffer between them filled up, producers would block — holding their threads — waiting for consumers to drain it. But consumers needed threads to run too. With all threads held by waiting producers, consumers couldn’t run. With consumers not running, Split Buffer couldn’t drain. With Split Buffer full, producers couldn’t unblock.
Get Ratul Dawar’s stories in your inbox
Join Medium for free to get updates from this writer.
When Split Buffer fills up, the producer holds its thread and waits for a “space is available” signal. That signal can only be sent once a consumer drains an item — but the consumer needs a thread to do that, and all threads are occupied. Nothing moves.
All threads are waiting for the queue to drain. Draining the queue requires running a callback. Running the callback requires a free thread. There are no free threads.
Perfect circle. Zero progress. This was reproducible every single time on any sufficiently large table — you just needed enough partitions to fill Split Buffer and trigger the condition. We confirmed it against Trino v455 and the master branch.
If you want to follow the investigation from the source, here is the original Trino issue we filed while debugging the incident.
The Fix: Stop Blocking, Start Yielding
The solution didn’t call for a ground-up rewrite. The fix was a change in how producers behave when Split Buffer is full: instead of blocking — holding the thread hostage while waiting — the producer now yields. It hands the thread back to the pool and registers a callback: “reschedule me when there’s room.” The thread is immediately free to do other work, including running the consumer that drains Split Buffer.
No thread is ever parked waiting. The pool stays free. The circular dependency simply cannot form.
Everything else — the file-listing logic, Split Buffer semantics, the overall architecture — stays the same. The change is surgical, isolated entirely to how the producer behaves when Split Buffer is full.
Resolution and Impact
The resolution was small in code but large in effect: producer tasks no longer monopolised the shared executor while waiting for space. The infinite planning wait disappeared because the consumer side could always get runtime to drain Split Buffer.
Takeaways
The incident was frustrating precisely because the system looked calm from the outside. CPU was quiet, memory was fine, logs were empty, and yet inventory replenishment jobs were stuck behind queries that would never finish planning.
First, idle resources do not always mean an idle system. Sometimes they mean every useful thread is waiting for work that cannot be scheduled.
Second, in shared executors, backpressure must not hold the scarce resource it needs someone else to release. If the system needs another task to make progress, give the thread back.
Third, yielding removes the deadlock, but it does not make large tables cheap to plan. Partition counts, scheduler fairness, and query shape still matter. Backpressure remains useful; it just stops being fatal.
If you’re excited about solving large-scale distributed systems problems, debugging the kind of failures that don’t show up in logs, and building infrastructure that powers real-world commerce — join us at Blinkit Tech.
Reach out at techhiring@blinkit.com
