The Evolution of Engineering Observability at Evolve
The Evolve platform was built more than a decade and a half ago. In 2024, we committed to modernize our platform for rapid innovation. As an engineer, there are many skillsets and tools required to modernize legacy monoliths but first and foremost is always…you got it, observability.
The key to supporting any tech stack is to have the ability to detect when things go awry, and then detect what things went awry. Without either, it takes much longer to know when a code deploy or configuration update has failed and how broad the impact of failure is. Time is of the essence.
As we kicked off our platform modernization and ramped up development over the past few years, Evolve’s engineering observability has transitioned from a fragmented, team-specific approach to a unified, metric-driven culture. This evolution has focused on reducing Mean Time to Detect (MTTD) and Resolution (MTTR) and more recently, we have also started integrating advanced tracing and AI-assisted diagnostics to manage the complexity of our integrations.
Phase 1: Establishing the Unified Baseline
Prior to May 2024, monitoring was siloed, leading to undetected bugs and customer-facing issues. Each team had their own set of practices for monitoring which varied wildly in capabilities which made it difficult to triage and solve issues without the SME. The initial development stage focused on standardization and tool selection.
We started a playbook of best practices and created a shared language for reliability across the organization. Following a three-month trial, DataDog was initially selected in April 2024 to manage logs, traces, and metrics. The choice was driven by the need for a comprehensive platform that could aggregate data from various legacy systems into a single pane of glass.
Early focus areas included moving our pricing pipeline and one of our high-traffic services to unified infrastructure with standardized monitoring. These critical path services served as the testing ground for the new observability standards before a wider rollout.
A “One Team, One Toolchain” approach was advocated to reduce cognitive load and maintenance burdens. By limiting the number of supported tools, the engineering org could focus on deep expertise rather than managing a fragmented ecosystem.
Phase 2: Tooling Pivot and the Observability Summit
As the scale of data grew, we evaluated cost vs. features, leading to a significant shift in the tooling strategy and the adoption of “Metrics > Logs” as a core philosophy.
By late 2024, we began migrating logging and metrics from DataDog to Sumo Logic to better manage noise and costs. This strategic shift allowed teams to consolidate their observability data into a more cost-effective platform while improving the signal-to-noise ratio in alerting.
In June 2025, we held an org-wide summit which emphasized that a healthy application should log very little and rely primarily on tagged metrics and automated alarms. The event served as a cultural turning point, aligning the engineering organization around the “Metrics > Logs” philosophy to streamline incident response.
The platform modernization (to be discussed in a separate Medium article, coming soon to a store near you!) took on several forms in the beginning but the core was a new platform built in Golang. This new monorepo had services which began utilizing Sumo Logic for traces, spans, and real-time dashboards. This implementation provided unprecedented visibility into the microservices architecture, allowing developers to visualize request flows and identify latency bottlenecks in real-time.
Get Nana Gapar’s stories in your inbox
Join Medium for free to get updates from this writer.
Adoption of ArgoCD also provided real-time pod status and console output, complementing traditional logging. By integrating deployment state with observability metrics, the team gained a holistic view of system health, making it easier to correlate software rollouts with performance changes.
Phase 3 (Today): Advanced Diagnostics and AI Integration
Today, we are focused on leveraging deep tracing and generative AI to resolve complex performance issues and automate operational health.
Teams across the organization have transitioned to using Sumo Logic traces for granular troubleshooting. This shift has enabled engineering squads to pinpoint specific points of failure within the microservices architecture, specifically targeting issues like JWT token expirations and suspicious user activity patterns that were previously difficult to isolate.
Recent advancements in our diagnostic workflows have integrated generative AI models, such as Claude, to parse complex trace data. By feeding trace logs into these models, engineers can receive specific, actionable code recommendations to mitigate memory spikes and prevent “out of memory” (OOM) kills before they impact production stability.
As of October 2025, our focus on observability has yielded a highly efficient lead time for changes, currently averaging just 10 minutes. However, addressing a 15% change failure rate remains a top priority; we are refining our monitoring strategies to provide better feedback loops and reduce regressions during the deployment process.
But wait, there’s more! Our observability strategy isn’t constrained to just our services, but it extends deeply into our external integrations to ensure end-to-end reliability. For our partners such as Airbnb, we have integrated with their Quality Statistics API to monitor listing performance in real-time. Similarly, we are moving our Salesforce integration toward a real-time observability model within Sumo Logic, which will replace the legacy “Exceptions” tables that were often too noisy to be useful for rapid incident response.
The Modern Tooling Landscape
The current observability stack is composed of several specialized tools. For logging and metrics, we use Sumo Logic as a unified platform for all new platform services. Tracing is handled via OpenTelemetry (OTEL), which provides auto-generated service maps and span analysis.
Dashboards are primarily managed through Sumo Logic and ArgoCD to monitor uptime gauges, HTTP request health, and pod status. Alerting is routed through Slack and Go Alert, with dedicated channels for teams’ pipeline failures. Incident management is also wired into our tools e.g., Sumo Logic monitors will send a post to a slack channel and also escalate to the on-call engineer.
The standardization of this stack and of focused efforts across the org has significantly reduced overall noise and enabled the teams to have faster MTTD and MTTR times.
Future Horizons and Strategic Planning
As part of our incident management and post-mortem process, we have deliberately focused on action items which enhance observability as part of our learnings. Following recent incidents, such as GraphQL outages and ECR image limit failures, the teams are committed to addressing all identified gaps. Future work will focus on implementing and validating enhancements to ArgoCD for better error checking during deployments, and improving server-side observability to prevent similar large-scale failures from recurring.
Ideally, we will have fully enabled tracing across 100% of our services. We will continue to develop and build towards this goal. Achieving full adoption is crucial for consistent and end-to-end troubleshooting, allowing engineers to reliably follow request paths across all microservices, leading to faster resolution times.
Ultimately, our commitment to technical rigor and automated feedback loops ensures that as the Evolve platform grows, our observability remains the heartbeat of a resilient and high-performing engineering organization.
