Zero Log Loss in Kubernetes: Why I Replaced a Single Fluentd DaemonSet with a Two-Layer Architecture
How separating log collection from log aggregation eliminated data loss during crashes, rolling updates, and infrastructure failures.
The Problem I Did Not Know I Had
Like most teams running microservices on Kubernetes, I started simple: one Fluentd pod per node, deployed as a DaemonSet, shipping container logs straight to Elasticsearch. It worked — until it didn’t.
The first sign was subtle. During a rolling update of the Fluentd DaemonSet itself, gaps appeared in my Elasticsearch indices: a few minutes of logs, simply gone. I dismissed it as a fluke. Then a node became unhealthy, Kubernetes evicted the Fluentd pod, and while it was rescheduling, another gap appeared. I added a counter metric to track the losses. It kept climbing.
By the time I decided to act, I had confirmed three distinct failure modes, each capable of losing logs permanently.
Three Ways a Single-Layer Pipeline Loses Logs
1. Memory buffer loss on pod restart. With the default in-memory buffer, any Fluentd crash — OOM, node pressure, a bad configuration update — discards everything in the buffer. No warning, no recovery, no replay.
2. hostPath buffer conflicts. Moving to a filesystem buffer on a hostPath volume fixed the memory problem and introduced a new one: during a rolling update, two Fluentd pods briefly coexist on the same node and fight over the same buffer directory. The result is corrupted chunks and dropped events.
3. No backpressure during downstream failures. When Elasticsearch was unavailable — routine maintenance, a network blip, a capacity event — Fluentd queued events in memory and started dropping them once the queue filled. There was no durable spillover and no way to push back on producers.
The root cause was the same in all three: Fluentd was doing too much. With collection, buffering, processing and delivery in a single component, a failure anywhere in the chain broke everything.
The Two-Layer Solution
The fix is conceptually simple: split collection from aggregation, and give each responsibility to a component purpose-built for it.
Layer 1 — Fluent Bit (DaemonSet). A lightweight C-based collector on every node. Its only jobs are to tail container log files, enrich events with Kubernetes metadata, and forward them downstream. It does not ship to Elasticsearch and it does no complex processing.
Layer 2 — Fluentd aggregator (StatefulSet). A centralised aggregation tier with persistent volumes. It receives events from all Fluent Bit instances, performs parsing and enrichment, manages durable file-based buffers, and delivers to Elasticsearch with retry logic and backpressure.
This separation contains failures. If Fluentd goes down, Fluent Bit buffers locally and retries indefinitely. If Elasticsearch goes down, Fluentd spills to its persistent volume and drains when connectivity resumes. Neither failure loses a single event.
Layer 1: Fluent Bit, Configured for Durability
Fluent Bit’s resource footprint — roughly 15 MB of memory per node — is its headline advantage. But for zero log loss, the critical configuration is persistence, not resources:
[SERVICE]
storage.path /var/log/fluentbit-buffer/
storage.sync normal
storage.backlog.mem_limit 50M[INPUT]
Name tail
Path /var/log/containers/*.log
storage.type filesystem
DB /var/log/fluentbit-buffer/tail.db
Mem_Buf_Limit 50MB[OUTPUT]
Name forward
Host fluentd-aggregator
Port 24224
Retry_Limit False
storage.total_limit_size 1G
Three decisions here are load-bearing:
storage.type filesystemmoves the input buffer from memory to disk. After a restart, Fluent Bit resumes exactly where it left off, using the SQLite database (tail.db) that tracks file positions.Retry_Limit Falsemeans Fluent Bit never gives up on delivering to Fluentd. When Fluentd is unavailable, events accumulate in the local filesystem buffer instead of being dropped.storage.total_limit_size 1Gcaps the local buffer so a long Fluentd outage can't fill the node disk. Combined withstorage.backlog.mem_limit, resource consumption stays bounded and predictable even during sustained failures.
Layer 2: Fluentd Aggregator, Configured for Zero Loss
The aggregator runs as a StatefulSet, not a Deployment, because StatefulSets give each replica a stable identity and — critically — a dedicated Persistent Volume Claim. That is what makes the buffer truly persistent.
<buffer>
@type file
path /var/log/fluent/es-buffer
flush_interval 1s
flush_thread_count 4
retry_forever true
retry_type exponential_backoff
chunk_limit_size 8MB
queue_limit_length 2048
total_limit_size 8GB
overflow_action block
</buffer>The most important setting is overflow_action block. When the buffer fills — because Elasticsearch is slow or unavailable — Fluentd does not drop events. It blocks the input plugin. That backpressure propagates upstream: Fluent Bit detects the stalled connection and activates its own local filesystem buffer. The entire pipeline stalls gracefully instead of losing data.
Get Alexandre Lourenço’s stories in your inbox
Join Medium for free to get updates from this writer.
retry_forever true with retry_type exponential_backoff keeps Fluentd retrying Elasticsearch indefinitely, backing off to avoid hammering a recovering cluster.
The 8 GB total_limit_size per replica, backed by a hyperdisk-balanced-rwo PVC on GCP, buys roughly 8 hours of sustained peak log volume before any data is at risk — more than enough time to respond to any realistic Elasticsearch incident.
High Availability: Anti-Affinity as a First-Class Concern
Two Fluentd replicas are not enough on their own. If both land on the same node and that node fails, both are lost. The fix is pod anti-affinity, enforced at the scheduler level:
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: app.kubernetes.io/name
operator: In
values: ["fluentd"]
- key: app.kubernetes.io/instance
operator: In
values: ["fluentd-aggregator-prod"]
topologyKey: kubernetes.io/hostnameThe choice of required over preferred is deliberate: I want the scheduler to refuse to co-locate replicas, even if that leaves one pending. A pending replica is visible. A replica silently placed on the wrong node — and then lost with it — is not.
What Happens During Common Failures
Fluentd rolling update. Fluent Bit detects the dropped TCP connection, activates its filesystem buffer, and queues locally. The new Fluentd pod comes up, the forward connection is re-established, and Fluent Bit drains. Log loss: zero.
Node failure (with a Fluentd replica on it). The surviving replica keeps receiving from all Fluent Bit instances. Kubernetes reschedules the failed replica, which re-mounts its PVC and resumes. Events buffered by Fluent Bit during the gap are delivered once it’s ready. Log loss: zero.
Elasticsearch maintenance window. Fluentd’s file buffer fills on the PVC; overflow_action block stalls the input; Fluent Bit buffers locally. When Elasticsearch returns, Fluentd drains first, then Fluent Bit. Log loss: zero.
Lessons Learned
Separate concerns early. A single DaemonSet is fine for small clusters with tolerance for occasional loss. The moment you need durability guarantees, the two-layer pattern pays for its complexity immediately.
File buffers are not free. Every node needs disk space for Fluent Bit’s buffer; every Fluentd replica needs a PVC provisioned in advance. Size them for your realistic failure scenarios, not your median throughput.
overflow_action block is the keystone. Without it, the backpressure chain breaks and Fluentd silently drops events under load instead of propagating the signal upstream.
StatefulSets demand more operational care. Scaling one down while it holds buffered data requires draining first. I added a pre-stop hook that waits for the buffer queue to empty before the pod is allowed to shut down.
Anti-affinity must be required, not preferred. A preferred policy gets violated under node pressure, defeating the purpose. A pending replica is always better than an undiscovered HA gap.
Technology Summary
Conclusion
This migration wasn’t driven by architectural elegance. It was driven by logs disappearing in production and the need to guarantee that never happened again.
The key insight is that collection and aggregation are different jobs with different reliability requirements. Fluent Bit excels at collection: fast, lightweight, with reliable filesystem-backed input buffers. Fluentd excels at aggregation: mature buffering, retry logic, and a rich output plugin ecosystem. Combined in two layers — persistent volumes at the aggregation tier and a backpressure chain connecting both — you get something a single component cannot offer: durability at every point of failure in the pipeline.
