Behind the Scenes: Optimizing Log Throughput in Kubernetes
How We Overcame Bottlenecks in Log Ingestion and Improved Performance by 4x Ahead of Max’s Launch in Asia-Pacific
Content
- Acknowledgments
- The Lay of the Land: An Architecture Overview
- The Issue: How we hit the limits for log throughput
- Resolution
- Conclusion
Acknowledgments
This troubleshooting and improvement wouldn’t have been possible without the drive and collaboration between my coworkers.
The Lay of the Land: An Architecture Overview
First, let’s define who’s who, and who does what.
Warner Bros. Discovery is a giant company; a behemoth put together by a series of mergers and acquisitions. I’ll spare you the details, Google will do a much better job than me at explaining the company's history. What we need to focus on is Max, the streaming platform offering high-end cinema productions, TV shows, documentaries, and sports. All in one place!
More specifically on the engineering front: Logging as a Service for Max.
At Max, LaaS (Logging as a Service) is an internal startup, a company within the company, that handles everything related to logging: whether it is a backend server running on Kubernetes, a serverless function emitting logs to AWS CloudWatch, or a log stream emitted to a Kafka topic by a third-party. The job of the LaaS platform is to make sure these logs end up in an OpenSearch dashboard ready to be queried by the service owners.
The Details
Let’s zoom in on Logging as a Service specifically for services running on Kubernetes.
When a service logs information to stdout or stderr (the default destination for most logging libraries), these logs are picked up and stored as text files by the container runtime of the Kubernetes environment.
By default, kubelet directs the container runtime to write logs into directories within /var/log/pods.
This allows the logs to still be accessible to the user via kubelet in case the container crashes or gets evicted.
More on the Kubernetes Logging Architecture.
To read and aggregate logs in a distributed system (also known as cluster-level logging), one of three approaches can be used:
- Read logs at the level of the nodes by having an agent collect logs on every node
- Run every application container with a dedicated sidecar that collects and sends logs to a backend
- Push logs directly from within an application to a backend
At Max, we use the first approach. We run Fluentd on every Kubernetes node to read, parse, and forward logs to the backend.
More specifically, Fluentd “tails” the log files (tail -f , see reference), then processes them through a series of parsers and filters.
These parsers and filters include, but are not limited to:
- Converting text logs to JSON, e.g. Nginx logs
- Extracting a timestamp, or assigning the current as a fallback
- Assigning runtime labels to the logs, e.g. Kubernetes metadata
Once this processing is done, Fluentd forwards the logs to multiple ingestion queues that will get the logs shipped to OpenSearch for querying and visualization.
The Issue: How we hit the limits for log throughput
This architecture worked pretty well when handling logs in the order of megabytes per second for thousands of distributed services, until a single service managed to log the same volume (several megabytes per second) but on a single host node — not so distributed anymore.
To understand why this is an issue, I’ll go on a tangent to explain how logs are retained on a host node, and how logs are flushed to a log ingestion queue.
Sub-issue 1: Log rotation by the container runtime
To prevent service logs from using excessive disk space, the container runtime is configured through kubelet to rotate log files.
This is how the log retention mechanism handles the lifecycle of log files:
Get Redouane Achouri’s stories in your inbox
Join Medium for free to get updates from this writer.
Log Rotation Mechanism
- The container runtime and kubelet (through the Container Runtime Interface) write service logs to an active log file
0.logthat is stored on the host node at:
/var/log/pods/<POD>/<CONTAINER>/0.log- When the size of the active log file reaches the value of
containerLogMaxSize(default 10Mi), this file is moved by renaming it to a timestamped file (e.g.0.log.20241216–152740) and a new active file0.logis created. - This process happens continuously, and up to
containerLogMaxFiles(default 5) log files are kept in storage. This ensures that several older logs are retained while disk space usage is kept under control.
More on the Kubernetes log rotation.
Sub-issue 2: Speed at which Fluentd flushes data to the queue
See, when Fluentd tails logs from a given service, it buffers the data in disk and flushes it periodically to one of the ingestion queues.
This is configured through an extensive set of parameters. The most relevant are:
refresh_interval: how often the list of files to tail (tail -f) is refreshed. This is relevant when the log files are rotated frequently.flush_interval: how often the buffer is flushed.flush_thread_count: how many threads are flushing simultaneously.chunk_limit_size: how big are the chunks to flush.
The Incident
For the preparation of Max’s launch in APAC, different teams load test their services to see how much load they can handle, e.g. how many simultaneous users can be streaming content.
One such test involved an API service that was running only 2 nodes and was generating over 2 MB/s on each node (read pod), however, only 25% of the logs were successfully captured to OpenSearch — meaning only 1 log out of 4 was ingested.
This concentration of all traffic on 2 nodes wasn’t intentional by the service team, but it allowed the logging as a service platform to unlock a higher throughput capacity. The following is how we optimized the Kubernetes cluster and the Fluentd configurations to achieve a higher capacity.
Resolution
After finding out about the container log rotation mechanism and its default value containerLogMaxSize of 10Mi, we realized that Fluentd’s refresh_interval was too large (meaning slow refresh) to allow Fluentd to pick up log files before they were rotated and archived.
Action 1: Increase the container runtime’s containerLogMaxSize from 10 Mi to 50Mi.
This allows the storage of more logs within a single log file, which results in less work trying to track and read every log file.
Action 2: Reduce the Fluentd refresh_interval from 60s to 5s.
This means that even if log files get filled and rotated after 50Mi, Fluentd can pick that rotation quicker to read the new log files. As a byproduct, we also have to increase the rotate_wait parameter which dictates how long Fluentd will continue to read from the older archived log files.
Launching a second load test, we noticed already a significant improvement in the log ingestion rate from 25% to 98%.
Already a progress, but we still need to ingest the missing 2% or at least explain why they’re dropped (e.g. due to explicit filtering).
And we also need to explain and resolve the high CPU usage that started appearing for Fluentd pods.
When performing a sigdump of the Fluentd process, we can see a large number of threads that are trying to flush the internal buffers.
When looking back at the configuration, we noticed that more than a dozen of the Fluentd output plugins had a high value for flush_thread_count — which explains why we could see so many threads in the sigdump — but also the flush_interval was set to a low value which caused very frequent attempts to flush the buffer, even when there was nothing to flush.
Did it click yet? A large number of threads working on flushing buffers too frequently is synonymous with a high workload, which translates to elevated CPU usage.
Once this usage reaches 100% of the CPU available, the Fluentd process starts getting throttled and thus it becomes unable to pick up and process logs, and send them to the ingestion pipeline.
Action 3: Reduce flush_thread_count from 30 threads to somewhere between 2 and 4 threads, and increase the flush_interval from 1s to 5s.
The immediate result was a drop in the CPU usage across all Fluentd pods:
This also resulted in the ingestion of 100% of the logs since the Fluentd processes were no longer throttled on high CPU usage.
Conclusion
Going through the challenge of understanding what caused bottlenecks in log ingestion, and optimizing the logging throughput taught us more than just tweaking configuration — it highlights the importance of understanding the relationship between infrastructure, workloads, and scalability.
By diving into Kubernetes logging mechanisms and Fluentd operations, we uncovered bottlenecks and implemented targeted fixes that boosted log ingestion rate by 4x or more — from 25% to 100% of 2Mb/s of logs — while reducing CPU usage.
Nobody knows why the parameters were given their value — passed from one generation of engineers to another, taken as dogma, and inherited from an era where the log rate was low. But as with anything, one must ask the question “Why?”.
Key lessons from this experience include:
- Adaptability: Logging configurations must evolve with changes in workload and scale. This means we must regularly review and test settings like
containerLogMaxSize,refresh_interval, andflush_thread_count. - Visibility: Monitoring is critical to uncover bottlenecks. Tools like sigdump and detailed metrics helped us pinpoint issues and validate improvements.
- Collaboration: Coordinating with service teams helped us identify unexpected workloads and unlock insights to optimize the system holistically.
- Elegant balance: Sufficient capacity to handle spikes, efficient resource usage, and clear visibility into system performance.
By continuously iterating and learning from challenges, engineering teams can not only prevent future incidents but also build robust foundations for innovation. Logging may be “behind the scenes,” but its impact is front and center for the success of any modern platform.
