Observability at Scale : Migrating FluentD from EC2 to EKS
How containerizing our logging pipeline transformed cost, reliability, and operational agility — Platform Engineering · BHN SRE Team · 2025
Executive Summary
Managing observability infrastructure across a large portfolio of enterprise platforms means that every inefficiency in the logging pipeline is magnified across regions, teams, and cost centers. In 2025, our Blackhawk Network’s (BHN’s) SRE and Platform Engineering teams completed a significant transformation: migrating our FluentD-based logging service from traditional EC2 instances to a fully containerized EKS (Elastic Kubernetes Service) architecture.
The results speak for themselves. Logging service resource costs dropped by approximately 83% through better bin-packing and right-sized pod limits. The broader infrastructure account shed roughly 30% of its EC2 instances. And EKS and EC2 infrastructure-specific incidents fell by approximately 80% year-over-year, with Critical and High priority failures seeing the steepest declines.
This post documents the full journey, covering the architectural decisions that preceded the migration, the phased execution approach, and the data that validates the outcome.
This migration was not a simple lift-and-shift. It was a deliberate re-architecture of how we think about log aggregation: moving from static, over-provisioned machines to elastic, self-healing, cost-efficient Kubernetes workloads.
Background and Architecture Context
FluentD acts as the central log router, responsible for ingesting, transforming, and fanning out log streams to destinations including Splunk, OpenSearch (S3-backed), and internal monitoring systems. Prior to this migration, these FluentD instances ran on standalone EC2 machines: manually provisioned, SSH-managed, and subject to the classic problems of static infrastructure.
The 2025 EKS migration sits within a broader multi-phase evolution:
Phase 1 (June 2024) was foundational: we decomposed a monolithic FluentD aggregator into four discrete microservices, namely logging-core, logging-os, logging-splunk, and logging-s3. This established the clean service boundaries that made the EKS migration tractable. Phase 2, the focus of this post, built directly on that foundation.
Phase 1 Recap: Microservice Decomposition
Before containerizing, we refactored. The original monolithic aggregator handled all log destinations in a single process, meaning a misconfigured Splunk forwarder could starve the S3 writer, or a buffer overflow in one pipeline would cascade across all others. Identifying and recovering from these cross-destination failures required significant manual investigation and often resulted in delayed alerting across multiple platforms.
Splitting into four independent services delivered the foundational properties that made the EKS migration viable:
- Fault Isolation: a buffer failure or plugin crash in one destination no longer cascades to others
- Independent Buffer Management: no more shared buffer contention on a single machine
- Deployment Independence: deploy changes to logging-splunk without touching logging-s3
- Independent Scalability: scale each service based on its own load profile
- Improved Code Ownership: clean boundaries, faster delivery, fewer conflicts
Decomposing before containerizing was the single most important prerequisite for the EKS migration. Containerizing a monolith creates a monolithic container, not a cloud-native workload.
Phase 2: EC2 to EKS Migration
Migration Timeline
The migration was executed in three phases to minimize risk and validate cluster behavior under real production load:
Running EC2 and EKS in parallel from May through September gave the team confidence before decommissioning the majority of the old fleet. The Active/Active EKS configuration across production and pre-production eliminates the environment drift that had previously caused class-specific failures. With both environments driven from the same Helm chart, what passes in pre-production is guaranteed to behave identically in production. A small EC2 presence has been retained to support legacy applications and platforms that are not yet containerized.
Key Architectural Improvements
Self-Healing and Resilience
- Automatic pod restarts eliminate manual EC2 recovery runbooks that previously required on-call engineer intervention
- Rolling updates replace risky stop/start sequences, keeping the logging pipeline active during upgrades
Elastic Scaling
- Horizontal Pod Autoscaler (HPA) scales logging pods during volume spikes, eliminating manual capacity planning around peak events
- Bin-packing on large worker nodes (16 vCPUs / 32 GB RAM) enables 8 to 12 logging-service pods per node
Security
- IRSA (IAM Roles for Service Accounts) enables pod-level IAM, removing the need for static access keys stored on EC2 instances and eliminating an entire class of credential rotation incidents
- Namespace isolation provides clean multi-tenancy with resource quotas, preventing any single consumer from starving others
Operational Tooling
- Unified GitOps deployment via Helm and manifests replaces SSH-based config pushes and manual AMI baking, making every change auditable and code-reviewed
- kubectl-driven diagnostics replace SSH-based manual recovery; issues that previously required 15 to 30 minutes of investigation are now diagnosed in seconds, dramatically reducing Mean Time to Resolution (MTTR)
Multi-Region Support
Get Blackhawk Network’s stories in your inbox
Join Medium for free to get updates from this writer.
The logging service now runs across three AWS regions, specifically us-west-2, us-east-1, and eu-west-1, with a planned expansion to the Sydney region. This multi-region footprint was impractical to maintain with EC2-based deployments. A single Helm chart with per-region value overrides now manages the entire fleet, making new region deployments a configuration change rather than an infrastructure project.
Resource Utilization and Cost Impact
Pod vs. EC2 Resource Comparison
With EKS bin-packing on 16 vCPU / 32 GB worker nodes, we can run 8 to 12 logging-service pods on a single EC2 node, compared to one service per EC2 instance in the old model. The migration reduces overall logging service resource costs by approximately 50%.
Cost Reduction
Migrating to EKS reduces overall logging service resource costs by approximately 50%, driven by improved bin-packing, right-sized pod limits, and elimination of idle over-provisioned EC2 capacity.
Beyond the logging service itself, consolidating workloads onto shared EKS worker nodes has contributed to a 30% reduction in EC2 instances across the infrastructure account. Additional cost levers include reduced inter-region data transfer costs from improved pod co-location, elimination of idle EC2 capacity that was over-provisioned to handle peak load, and significantly reduced on-call engineering time as self-healing infrastructure handles failures that previously required manual intervention. The migration also removed the overhead of maintaining AMI pipelines, instance patching schedules, and per-host SSH key management.
Incident Analysis: Before and After
The clearest measure of architectural improvement is incident reduction. Tracking incidents directly attributed to EKS and EC2 logging infrastructure across the five quarters from Q1 2025 through Q1 2026 tells a compelling story about the impact of the migration.
Infrastructure incidents were at their highest pre-migration, then fluctuated during the in-flight period when EC2 and EKS systems ran in parallel and both platforms contributed to the incident pool. The parallel-run phase was intentional: it allowed the team to validate EKS behavior under real production load before committing to the primary cutover.
The trade-off was a temporary increase in combined incident surface. Once the majority of the EC2 fleet was decommissioned in September 2025, incidents dropped sharply and stabilized well below the baseline. The year-over-year comparison from Q1 2025 to Q1 2026 reflects an approximately 80% reduction in EKS and EC2 infrastructure-specific logging incidents.
That reduction does not happen by accident. The architecture directly eliminates the failure modes that drove the pre-migration baseline:
- Buffer overflow cascades: eliminated by microservice isolation (Phase 1) and per-pod resource limits
- Host-level failures requiring manual recovery: eliminated by Kubernetes self-healing and automatic pod restarts
- Config drift between environments: eliminated by GitOps and Helm-based deployments with code-reviewed manifests
- Scaling lag during peak log volumes: addressed by HPA auto-scaling responding to real-time load
- IAM key rotation incidents: eliminated by IRSA pod-level roles replacing static EC2 credentials
The ~80% infrastructure incident reduction is particularly significant because it was achieved while platform coverage and log volume expanded. The absolute workload on the logging pipeline increased, yet this category of failure was drastically reduced.
What’s Next
- Expanding the logging service to another AWS region to support Pacific workloads with local log aggregation
- Quarterly Incident Trending: continuing to track logging-related incident rates across quarters to validate sustained improvement post-migration
- VPA Evaluation: exploring Vertical Pod Autoscaler alongside HPA to further right-size resource requests based on observed usage patterns
- Logging Service Dashboard: publishing a dedicated observability dashboard surfacing pod health, throughput per destination, HPA scaling events, and inter-region transfer metrics
Lessons Learned
Decompose before you containerize. The Phase 1 microservice split was the single most important prerequisite. Four well-scoped services containerize cleanly; a monolith containerizes as a monolith and inherits all the same failure modes in a new runtime.
Phase the risk, not just the timeline. Running EC2 and EKS in parallel for several months before decommissioning the majority of the legacy fleet was the right call. The short-term cost of operating both environments was justified by the risk reduction and the confidence it provided. Retaining a small EC2 footprint for legacy platforms also meant no applications were forced into a migration they were not ready for.
GitOps pays dividends immediately. Moving from SSH-based config pushes to Helm and manifest-driven deployments required upfront investment, but the elimination of config drift and the ability to do code-reviewed, auditable deployments was felt within the first sprint post-migration.
Right-size at the pod level, not the instance level. EC2 encourages over-provisioning because you pay for the whole instance regardless of utilization. Kubernetes encourages precision: accurate resource requests and limits let the scheduler handle bin-packing, and the economics follow naturally.
Conclusion
The FluentD EC2-to-EKS migration delivered results across every dimension that matters: cost, reliability, security, and operational efficiency. What started as a desire to reduce infrastructure spend became a comprehensive rethink of how the team manages observability at scale.
The microservice decomposition gave clean service boundaries and eliminated cascading failures. The Kubernetes migration gave elasticity, self-healing, and a GitOps-driven deployment model that treats infrastructure configuration as first-class code. Multi-region expansion gave global reach without proportional operational overhead. And the OpenSearch additions gave demanding platforms dedicated capacity without affecting shared workloads.
The incident data now backs what the architecture promised: fewer incidents, lower severity when they do occur, and a logging infrastructure that scales with the business rather than against it. The parallel reduction in EC2 cost and the sharp drop in Critical and High priority incidents in Q1 2026 are not coincidences. They are the direct and measurable result of deliberate architectural investment.
Author — Adir Meidan, with Kaushik Kumar as the lead engineer who drove this project under the guidance of Distinguished Engineer Michael Lasevich. This blog is grounded in data and metrics, reflecting work conducted at Blackhawk Network.
