<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Pranavi Chandramohan on Medium]]></title>
        <description><![CDATA[Stories by Pranavi Chandramohan on Medium]]></description>
        <link>https://medium.com/@pc689?source=rss-ff7573cbe5f0------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*OMlYMhkQKZxpdEsf.</url>
            <title>Stories by Pranavi Chandramohan on Medium</title>
            <link>https://medium.com/@pc689?source=rss-ff7573cbe5f0------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 15:54:55 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@pc689/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Observability at Campaign Monitor]]></title>
            <link>https://medium.com/cm-engineering/observability-at-campaign-monitor-22796334958?source=rss-ff7573cbe5f0------2</link>
            <guid isPermaLink="false">https://medium.com/p/22796334958</guid>
            <category><![CDATA[observability]]></category>
            <category><![CDATA[datadog]]></category>
            <category><![CDATA[apm]]></category>
            <category><![CDATA[site-reliability-engineer]]></category>
            <dc:creator><![CDATA[Pranavi Chandramohan]]></dc:creator>
            <pubDate>Wed, 15 Jul 2020 23:12:03 GMT</pubDate>
            <atom:updated>2020-07-15T23:12:03.532Z</atom:updated>
            <content:encoded><![CDATA[<p>These are exciting times for engineers at CM as many features from our core app (a monolith) are being refactored into microservices. A system which had a request path like below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/468/1*c8X9nKaEce1b5MRV8D0XjQ.jpeg" /></figure><p>now looks like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/494/1*_UjDpVVn5s9D7CoihCIr6g.jpeg" /></figure><p>When a system starts to scale in terms of the number of microservices, mapping a request’s end to end journey becomes harder. Unless you have strict documentation procedures, maintaining architecture diagrams in a wiki is hard as they can become outdated quickly and are then more misleading than useful.</p><p>For the Site Reliability Engineering (SRE) team who are responsible for the reliability of the platform and emergency response, this means more complexity and chaos.</p><p>I joined the SRE team at a time where engineers with the most knowledge about the monolith had moved on, so not only did we have to support the newly added microservices but also the giant monolith! We realised that we needed the system to answer a very important question: <em>what is broken and why?</em></p><p>For the last year, our main goal has been to make our system answer this question. In this post, we will look at the ways we went about accomplishing this.</p><h4>What is observability?</h4><blockquote>Observability is a measure of how well internal states of a system can be inferred by knowledge of its external outputs. — wikipedia</blockquote><p>If we can answer questions about an application without having prior knowledge about it, then we can say that the application is observable. This is the key to helping us resolve production issues for our system. To make it observable, the app has to be instrumented to expose relevant data.<br>Let’s go through all the different kind of data we expose from our application.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*7HuFR1r3TNiIfxC15XOIsg.jpeg" /><figcaption>Source: <a href="https://www.elastic.co/blog/observability-with-the-elastic-stack">https://www.elastic.co/blog/observability-with-the-elastic-stack</a></figcaption></figure><h3>Logs</h3><p>Extensive logging has always been practised by developers at CM but, without structure, it is hard to find and analyze relevant logs. We live in an age of distributed microservices and unstructured logs just don’t cut it. Logs need structure, and by that I mean they need fields and values so that we can search and aggregate on those field values. There are libraries we use to provide structured logging for eg SLF4J for Java, Serilog for C# and Zerolog for GO.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*R7XfdI2GL_KRg75q4AWO-Q.png" /></figure><p>It is not possible to change all the applications at once but every time we touch a part of a system we ensure the right logging libraries and right formats are used. Organically our log data has become significantly more useful!</p><h4>Tools</h4><p>We send all our application logs to <a href="https://www.graylog.org/">Graylog</a> which stores the data in an Elasticsearch (ES) cluster. Graylog is quite a powerful tool with options to manage ES indexes, alerting, dashboards and many more cool features.</p><h3>Metrics</h3><p>Our engineers use metrics heavily throughout the application. Different metric types like counts, histograms and gauges are used to give information about a particular process or activity over time. However, these were not standardised.</p><p>Every application has inputs and outputs like: Amazon Simple Queue Service (SQS), RabbitMQ, Kafka, HTTP/gRPC endpoints etc via which data is ingested, processed and passed on to another application or stored for persistence. For each of these inputs and outputs, it is important to measure all the key metrics. This will determine if the issues are with the application itself or its dependencies.</p><p>What are the key metrics we look at?</p><ul><li>Traffic / Usage</li><li>Latency</li><li>Errors</li><li>Saturation of resources (CPU / Memory/ IO)</li></ul><p>An application could use many services like gRPC, Kafka, Redis etc. In order to standardise the metrics we collected for each of these services, we created a list of key metrics. We verified that every application or feature which is ready to be released exposed these key metrics. This gave the team confidence in being able to find the right metrics to help troubleshoot the issue.</p><h4>Tools</h4><p>We use <a href="https://docs.datadoghq.com/metrics/">Datadog</a> for our metrics and monitors and PagerDuty for event management. We integrated Datadog monitors with PagerDuty, to ensure any metric breaching threshold(s) alerts an engineer.</p><h3>Why not just be happy with logs and metrics?</h3><p>Logs and metrics are great tools for observability. However, they have gaps like:</p><ul><li>With logs, we often struggle with not knowing what log messages to search for (due to the sheer volume of log data).</li><li>With metrics, the data is aggregated so it is helpful for a birds-eye view but not to drill down to specific parameters.</li><li>With Datadog, adding <a href="https://docs.datadoghq.com/developers/metrics/">custom metrics</a> helps in performing deeper analyses but that comes with a huge price tag <strong>$$$$</strong></li></ul><p>We add metrics and logs where we think something might go wrong but what about the unknown unknowns? What about understanding the end to end journey of a request?</p><h3>Traces</h3><p>Datadog’s <a href="https://docs.datadoghq.com/tracing/">APM</a> has been a game-changer for us in terms of observability.<br>We started with a basic set up following the guidelines <a href="https://docs.datadoghq.com/tracing/setup/dotnet-framework/?tab=environmentvariables">here</a>. With just the initial traces for HTTP requests, SQL queries and Redis operations we were able to derive more insights into our application. The visual appeal of tracing for me was a clear win over logs and metrics.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lpaXVI4lNBAQT-azMYx52Q.png" /><figcaption>Source: <a href="https://docs.datadoghq.com/tracing/visualization/">https://docs.datadoghq.com/tracing/visualization/</a></figcaption></figure><p>APM made the dependencies between various services visible with <a href="https://docs.datadoghq.com/tracing/visualization/services_map/"><em>services map</em></a>. <em>Watchdog</em> is another cool feature that will automatically detect anomalies in traces and provide alerts on system performance. Datadog also provides <a href="https://docs.datadoghq.com/tracing/guide/metrics_namespace/">Trace Metrics</a>, which are the following aggregate statistics over all the traces instrumented:</p><ul><li>Total requests and requests per second</li><li>Total errors and errors per second</li><li>Latency</li><li>Breakdown of time spent by service/type</li><li><a href="https://docs.datadoghq.com/tracing/faq/how-to-configure-an-apdex-for-your-traces-with-datadog-apm/">Apdex score</a> (web services only)</li></ul><p>In order to customise some of the functionality, we created our own wrapper around Datadog’s library for each language we support: C#, Java and Go. This gave us more control over creating traces and how we wanted to serialise the span context. We trace distributed calls by adding the span context (<em>traceId</em> and <em>spanId</em>) as a header value from the client. In the server, we could then use that span context as the parent giving visibility into the journey of a request across microservices.</p><p>Datadog samples the traces that are stored in the system. It is important to understand the <a href="https://docs.datadoghq.com/tracing/guide/trace_sampling_and_storage/?tab=java">sampling rules and storage</a> of traces in APM to help set the right sampling priority, especially when dealing with distributed tracing.</p><h3>Baking in APM</h3><p>Tracing does involve adding more code to the application, this could be a blocker for widespread adoption. In order to make it easier for our engineers, we created libraries that are easy to plug into existing services and in some cases, we added tracing in existing libraries. As of now, services like MSSQL, SQS, Couchbase, Cassandra, Kafka, Redis, gRPC are traced. This has significantly increased our visibility into the performance of our entire system.</p><h3>Conclusion</h3><p>Datadog has been key to our observability journey. We did consider Newrelic, Honeycomb and a few other options but Datadog has achieved a happy medium in terms of features and cost.</p><p>Knowing that if we get paged in the middle of the night, we have the right information to deal with the issue has helped us sleep better ;)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=22796334958" width="1" height="1" alt=""><hr><p><a href="https://medium.com/cm-engineering/observability-at-campaign-monitor-22796334958">Observability at Campaign Monitor</a> was originally published in <a href="https://medium.com/cm-engineering">Campaign Monitor Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>