Clickhouse: The Cache Stampede problem
Imagine you’re building an observability platform that currently supports 5 concurrent users. Now, as part of a scaling initiative, you’re asked to support 100 concurrent users instead.
Sounds like a straightforward capacity-planning exercise — until you realize the database itself is the bottleneck.
The resource nightmare
If your database needed x resources to serve 5 concurrent users, basic math says resources need to grow roughly 20x to serve 100. That’s a nightmare in any environment, but it’s especially brutal in resource-constrained, on-prem setups — and doubly so when the database powering your dashboards is a beast like ClickHouse.
So you do what any reasonable engineer would do: you go spelunking through your query patterns. What is your platform actually asking the database to compute every time it renders one of those beautiful graphs for your customers?
That’s when you notice something encouraging: across users, the queries are largely the same. The result of one user’s query can, in many cases, be reused for another user’s identical request. It’s a classic software engineering problem — and a classic software engineering solution: a shared cache.
Enter ClickHouse’s native query cache
The good news is you don’t have to build this cache yourself. ClickHouse has query caching built in natively. It lets you share computation across users and, in theory, keeps database resource consumption growing linearly with the number of users instead of multiplying out of control.
You run it through its paces, most cases check out, and it looks like the fix you needed.
But there’s one dangerous edge case waiting to turn your happy customers into grumpy ones.
What is a cache stampede?
A cache stampede (also known as a thundering herd) is a system failure that happens when multiple concurrent requests simultaneously experience a cache miss on the same popular item — and all of them turn around and hit the underlying database at the exact same moment.
Get Pranav Mehta’s stories in your inbox
Join Medium for free to get updates from this writer.
Here’s the sequence:
- A popular query result is cached.
- Its TTL expires.
- A wave of concurrent, identical queries all miss the cache at once.
- Every single one of them independently recomputes the result and pounds the database simultaneously.
A single expired cache key can be enough to bring down your entire system.
Where ClickHouse’s query cache falls short today
Right now, ClickHouse’s query cache is susceptible to exactly this problem. Here’s what happens under the hood:
- The same query runs repeatedly, and the first execution’s result gets cached.
- When the TTL expires, the entry goes stale.
- Every new (still identical) query that comes in now sees a cache miss.
- All of them recompute the result independently, and the first one to finish writes it back to the cache.
In an ideal world, only the first query after expiry would actually hit the database — recompute the result, refresh the cache — while every other identical query in flight would simply wait for that result and reuse it. Instead, ClickHouse currently lets all of them stampede the database at once.
The good news: there’s already an open PR working to solve this within a single ClickHouse node.
But wait — there’s a cluster-sized catch
Remember why we started this whole exercise? We’re not scaling a single node — we’re scaling an entire observability platform. That means the fix can’t stop at single-node caching. It needs to extend to a multi-node ClickHouse cluster.
To that end, ClickHouse is aiming to extend the cache stampede optimization across the cluster by implementing a Redis-backed query cache, enabling cross-node cache sharing. Instead of each node maintaining its own isolated cache (and its own stampede risk), nodes would be able to share cached results, closing the gap that single-node fixes leave open.
Where things stand
If you’ve run into this problem in ClickHouse, you’re not alone — the ClickHouse team is well aware of it. Work is actively in progress on both the single-node and cluster-wide fronts.
Until then, if you want your database to solve this for you automatically… pray to the gods of entropy to keep your platform’s users enchanted under the spell of disorder, so that the evil of synchronicity is kept at bay. 😄
Special thanks to Shiv Jha for highlighting the cache stampede problem and community work around it in Clickhouse.
If you want more posts like this on what’s coming in ClickHouse, follow me for more.
