Surviving Production: The Performance Gauntlet
OOMKills, probe misconfigurations, and connection pool storms — the real story of shipping a new architecture
Part 2 of: Beyond REST — Rebuilding an Eventually Consistent Discovery Platform
Quick Context
In Part 1, we covered how the Market Discovery Team at Getir transformed its architecture: replacing synchronous REST calls with an event-driven data mesh, denormalizing MongoDB for read-optimized materialized views, and consolidating multiple Kubernetes deployments into a single domain-agnostic service. Category page load times dropped from 7–8 seconds to ~2 seconds (P90).
Everything looked great in testing. Then we hit production traffic.
When Reality Hit
We had speed. Now we needed stability.
As we tuned resource requests to reduce costs, problems surfaced:
- Pod crashes: Traffic spikes caused OOMKills and CPU throttling. No headroom for burst handling.
- Karpenter instability: Node recycling caused prolonged pod startup times. JVM warm-up plus aggressive readiness probes meant pods were killed before they could initialize.
- Rolling deployment failures: New pods failed readiness checks under production traffic. Deployments stalled.
- Orchestrator timeouts: TCP sockets closed before backend responses completed.
- Database CPU throttling: MongoDB instances throttled under sustained query load. Connection storms amplified the problem.
Root Cause Analysis
We went back to fundamentals. Systematic investigation revealed five interrelated root causes:
1-) Incorrect Readiness Probes
Simple HTTP health checks didn’t account for JVM class loading, connection pool warm-up, or cache population. Pods were marked ready before they could actually handle production throughput.
2-) Liveness Probe Interference
The liveness probe shared the same thread pool as the application. Under high CPU load, it couldn’t respond within timeout. Kubernetes killed healthy but saturated pods.
3-) Connection Pool Exhaustion
Default pool sizes for HTTP clients and MongoDB driver were sized for the old architecture. The unified service’s dramatically higher throughput exhausted pools. Requests queued, latency inflated, timeouts triggered.
4-) Unoptimized Memory
Category detail endpoints loaded entire product collections into memory without pagination. Large heap allocations triggered frequent GC cycles, including Stop-The-World pauses that increased OOMKill risk.
5-) CPU-Intensive Serialization
The codec for Kafka and HTTP was CPU-heavy. Large binary payloads buffered entirely in memory before processing created sustained CPU spikes.
The Solutions
We systematically fixed every issue.
Memory Optimization — Concurrent Subcategory Pagination
Instead of fetching all products in one unpaginated query, we introduced server-side pagination at the subcategory level. The orchestrator breaks category detail requests into multiple concurrent subcategory-scoped paginated queries and assembles the response.
Result: Response times dropped to ~2 seconds (P90). Heap utilization decreased visibly. GC pressure reduced significantly.
CPU Optimization — Efficient Codec & Stream Processing
We replaced the serialization codec with a CPU-efficient alternative. Large binary arrays switched from in-memory buffering to stream-based processing. This eliminated memory spikes and spread CPU load evenly.
Result: Measurable reduction in GC Stop-The-World pause frequency and duration.
Kubernetes Probe Reconfiguration
Three critical changes:
Startup Probe: Added generous initialization window covering JVM warm-up, cache hydration, connection pool initialization. Resolved rolling deployment stalls.
Readiness via Actuator: Reconfigured to use Spring Boot Actuator’s composite health indicator. Validates DB connectivity and Kafka consumer status. Pods only marked ready when they can genuinely serve traffic.
Liveness Isolation: Moved liveness endpoint to dedicated management port with isolated thread pool. Even when application threads are saturated, liveness responds independently. Eliminated false-positive pod kills.
Get berkay özdemir’s stories in your inbox
Join Medium for free to get updates from this writer.
Spring Boot management configuration:
management:
endpoints:
web:
base-path: "/"
exposure:
include:
- health
- liveness
- readiness
server:
port: ${MANAGEMENT_PORT:9090}HTTP Connection Pool Tuning
Adjusted Keep-Alive to maintain persistent connections, tightened idle connection eviction to prevent zombie connections, shortened connection TTL to rotate across backend pods naturally for better load distribution.
Database Connection Pool Tuning
Increased maximum pool size proportionally to throughput expectations, added connection ramp-up rate limiter to prevent burst creation, configured warm pool maintenance, tuned wait queue to fail fast instead of queueing indefinitely. Improved tail-latency behavior under saturation.
The Results: A Platform Transformed
Today, our Discovery platform is unrecognizable from where we started:
- Category pages load in approximately 2 seconds(P90) (down from 7–8 seconds)
- Zero cascading failures from upstream service outages
- Significantly reduced infrastructure costs through consolidation
- Stable pod behavior under traffic spikes
- Successful rolling deployments without manual intervention
- Dramatically higher throughput with better reliability
The original 7–8 second figure came directly from Firebase mobile TTI (Time to Interactive) measurements. From a user perspective, category pages felt slow and unresponsive.
Our analysis showed that the latency was caused by the relational-style design in MongoDB. Although we were using a NoSQL database, the data model was heavily normalized, forcing cross-collection lookups and in-memory assembly at query time. A single Category Detail request previously triggered up to 9 concurrent database queries, which were then aggregated in application memory — amplifying latency under load.
It’s important to clarify what did not change:
- There was no meaningful improvement or degradation in inter-service communication
- There was no significant change in serialization/deserialization (serde) costs
The performance gain came almost entirely from redesigning the data model. By moving to denormalized, materialized views optimized for the read path, we eliminated runtime joins and reduced database calls from nine to 1–2 optimized queries per request. Backend P90 latency dropped to ~2 seconds.
Because mobile TTI was largely driven by backend response time, this translated directly to the client: Firebase TTI decreased from 7–8 seconds to approximately 2 seconds (P90).
In short, we didn’t optimize the network or the app layer — we fixed the data model and simplified the database access pattern.
The Trade-offs: What We Accepted
No solution is perfect. We made deliberate trade-offs:
Eventual Consistency: Price and stock changes take seconds to propagate through Kafka. For category browsing, this is acceptable. We monitor consumer lag closely and alert on propagation stalls.
Data Duplication: Denormalization means data lives in multiple places. The Discovery DB is a derived, read-optimized projection. Schema changes to upstream data products require corresponding updates to materialization logic.
Single-Service Risk: Consolidating all domain types into one service means a deployment bug affects all domains simultaneously. We mitigate this with startup probes, canary deployments, and robust readiness checks.
Orchestrator Dependency: The socket-level timeout issue is an external dependency. We’ve mitigated but not fully resolved it. Coordination with the orchestrator team continues.
What’s Next: The Road Ahead
We’re not done. Here’s what we’re considering:
- Client-side pagination: Reduce orchestrator assembly overhead, enable lazy-loading UX patterns
- Orchestrator timeout alignment: Eliminate socket termination for deep backend calls
- GraalVM native image: Reduce JVM warm-up time, improve pod startup characteristics
- Cache layer: Improve Redis usages and document sizes for hot category paths to push sub-second latency at the edge
- Distributed tracing: OpenTelemetry to correlate end-to-end latency from client through orchestrator, Discovery service, and database
- Resource Optimization Part 2: We decreased used pod count drastically, however still using a little more resources than expected.
The Bottom Line
When users load a category page today, they see products in about 2 seconds (P90). They don’t see the Kafka streams, the denormalized database, the carefully tuned connection pools, or the isolated health checks. They don’t need to.
They just see a fast, reliable shopping experience. And that’s exactly what we built it for.
Sometimes, you need to rebuild the foundation to build something that lasts. This was one of those times.
