I Almost Lost Log Data During a Restart — Here’s What Splunk Persistent Queues Taught Me
A few weeks ago, I restarted a Splunk indexer during a busy afternoon.
Nothing dramatic ,just a routine configuration change that required a restart to take effect. I kicked off the restart, waited for the service to come back online, and moved on with my day.
A few hours later, while reviewing dashboards, something felt off.
There was a small gap in the data right around the restart window. Not a massive outage. Not an obvious failure. Just a few missing events enough to notice if you were paying attention.
Unfortunately, those missing events happened during a critical incident upstream.
That sent me down a rabbit hole.
Where did the data go? and importantly
How do you make sure it never happens again?
That investigation led me to one of Splunk’s most overlooked reliability features: Persistent Queues (PQ).
Here’s the explanation I wish someone had given me when I first started working with Splunk.
😺What Really Happens When Splunk Restarts?
Like many people, I assumed Splunk immediately writes every incoming event to disk.
That’s not actually how the ingestion pipeline works.
Before data gets indexed, it travels through a series of internal queues. Think of them as temporary waiting rooms where events are held while Splunk parses, timestamps, and processes them.
The important detail?
Those queues are in memory by default.
That means queued events live only in RAM.
If Splunk is restarted, crashes unexpectedly, or the process is terminated while events are sitting in those queues, that data may never make it to disk.
It’s not delayed.
It’s not waiting to be indexed later.
It’s simply gone.
In my case, the restart happened at exactly the wrong moment. Events were sitting in memory when the service stopped, and those events disappeared along with the process.
🤠The Safety Net: Persistent Queues
This is where Persistent Queues come in.
Once I understood the concept, it felt surprisingly obvious.
Instead of keeping queued data only in memory, Splunk can write queued events to disk.
So if Splunk crashes or restarts unexpectedly, the queued data survives.
When the service starts again, Splunk reads the queue from disk and continues processing from where it left off.
A simple analogy:
- Memory queue: Notes written on a whiteboard
- Persistent queue: Notes written on paper
If the power goes out, the paper is still there.
The whiteboard might not be.
Persistent queues essentially provide a layer of durability between data ingestion and indexing.
😺Where Do You Enable Persistent Queues?
One thing that initially confused me is that Persistent Queues are not a single global setting.
The configuration depends on which part of the data pipeline you’re trying to protect.
😺On Universal or Heavy Forwarders
If the concern is protecting data when a forwarder temporarily loses connectivity to downstream indexers, configure queueing in outputs.conf.
[tcpout:my_indexers]
server = indexer1:9997, indexer2:9997
maxQueueSize = 500MB
This allows the forwarder to retain data locally until connectivity is restored.
😺On Indexers or Heavy Forwarders
If the concern is local ingestion pressure for example, data arriving faster than it can be parsed and indexed you can configure persistent queues on inputs.
[monitor:///var/log/myapp/*.log]
persistentQueueSize = 1GB
In this scenario, incoming data is written to disk-backed queues before being fully processed by the indexing pipeline.
✌️😉The Part That Surprised Me
Persistent Queueing is not enabled by default.
Splunk doesn’t automatically assume we want disk-backed protection for every workload.
We must explicitly configure it.
That’s easy to overlook because everything appears fine until the day a restart, outage, or crash occurs at exactly the wrong time.
😁Lessons I Learned the Hard Way
1. Persistent Queues Need Real Disk Space
Queue files live on disk, typically under:
$SPLUNK_HOME/var/run/splunk
That means enabling Persistent Queues isn’t free.
If you’re allocating a 10 GB queue, Splunk needs 10 GB available when the queue fills.
Get Uniqueinfo’s stories in your inbox
Join Medium for free to get updates from this writer.
Always factor storage requirements into the capacity planning.
2. Undersized Queues Create a False Sense of Security
A queue that’s too small isn’t much better than no queue at all.
If the queue fills before downstream services recover, ingestion pressure returns and data can still be blocked or dropped.
A better sizing question is:
How much data could be generated during my worst realistic outage?
For example:
- 5 GB/hour ingestion rate
- Potential 4-hour indexer outage
Required queue size:
5 GB × 4 hours = 20 GB
Start with outage scenarios rather than arbitrary sizes.
3. There’s a Performance Trade-Off
Persistent Queues introduce additional disk I/O.
Instead of temporarily holding events in RAM, Splunk continually writes queue data to storage.
For most environments, the overhead is acceptable.
But on heavily utilized systems, it’s worth monitoring disk performance after enabling the feature.
Reliability almost always wins over marginal performance gains — but understanding the trade-off matters.
😺How I Verify Persistent Queues Are Working
After enabling Persistent Queues, I wanted proof they were actually doing something.
These are the checks I now use.
Monitor Queue Metrics
Splunk exposes queue statistics through internal metrics logs.
index=_internal source=*metrics.log group=queue
| stats avg(current_size) avg(max_size) by name
Pay attention to queues that consistently approach their maximum size.
A queue running near capacity is usually a symptom of an ingestion bottleneck somewhere else in the pipeline.
Use the Monitoring Console
For a quick health check, the Monitoring Console is often the fastest option.
Navigate to:
Monitoring Console → Indexing Performance
We can easily visualize queue fill levels and identify bottlenecks without writing custom SPL searches.
This became my preferred approach when I just needed a quick operational snapshot.
🤷♀Look for Queue Files on Disk
Sometimes the simplest validation is the best one.
During a backlog or connectivity issue, check the persistent queue directory.
If persistent queueing is enabled, Splunk creates queue files on disk. For TCP output queues, check:
$SPLUNK_HOME/var/run/splunk/tcpout/
We can verify their existence with:
ls -lh $SPLUNK_HOME/var/run/splunk/tcpout/
Seeing queue files here during a backlog is a quick confirmation that Persistent Queues are working.
If Persistent Queues are enabled and actively being used, we should see queue files growing on disk.
If we don’t see any files:
- Verify the configuration was deployed correctly.
- Confirm the app containing the configuration is active.
- Ensure Splunk was restarted after the change.
- Double-check the stanza and file placement.
I’ve seen perfectly valid configurations fail simply because they ended up in the wrong app’s local directory.
🙂What I’d Tell My Past Self
Persistent Queues are one of those features you rarely think about until the exact moment they become critical.
They’re invisible when everything is healthy.
But during a:
- Restart
- Crash
- Indexer outage
- Network interruption
- Parsing backlog
they can be the difference between recovering gracefully and permanently losing data.
My missing events weren’t catastrophic.
They didn’t bring down the environment.
But they were enough to remind me that data reliability isn’t something you notice when it works ,it’s something you notice when it doesn’t.
If your organization cares about data completeness, spend twenty minutes today checking whether Persistent Queues are configured on your forwarders and indexers.
It’s a small investment that can save you from a very uncomfortable conversation later.
Have you ever had a “Wait… where did that data go?” moment in Splunk? I’d love to hear your experience and what you learned from it. 💕
