<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Subin T P on Medium]]></title>
        <description><![CDATA[Stories by Subin T P on Medium]]></description>
        <link>https://medium.com/@subinsundar?source=rss-c85f92e3fa0e------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*tIbqm-yhT0hpdpw0.jpeg</url>
            <title>Stories by Subin T P on Medium</title>
            <link>https://medium.com/@subinsundar?source=rss-c85f92e3fa0e------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 04:51:35 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@subinsundar/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Tracing our Observability Journey]]></title>
            <link>https://engineering.razorpay.com/tracing-our-observability-journey-a3c72f52d77f?source=rss-c85f92e3fa0e------2</link>
            <guid isPermaLink="false">https://medium.com/p/a3c72f52d77f</guid>
            <category><![CDATA[fintech]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <category><![CDATA[monitoring]]></category>
            <category><![CDATA[observability]]></category>
            <category><![CDATA[distributed-tracing]]></category>
            <dc:creator><![CDATA[Subin T P]]></dc:creator>
            <pubDate>Fri, 11 Feb 2022 13:36:06 GMT</pubDate>
            <atom:updated>2022-02-11T14:24:06.434Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Authors: </em><a href="https://medium.com/@venkatv"><em>Venkat Vaidhyanathan</em></a><em> and </em><a href="https://medium.com/@subinsundar"><em>Subin TP</em></a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qo82pp8QBMARLw9kyuqDuA.png" /></figure><blockquote>“Are you watching closely?” <br>— Robert Angier (From the movie “Prestige”)</blockquote><p>In the articles <a href="https://engineering.razorpay.com/scaling-to-trillions-of-metric-data-points-f569a5b654f2">Scaling Metrics</a> and <a href="https://engineering.razorpay.com/tracing-100k-spans-with-razorpay-opencensus-php-fork-d217aa9eca6">Tracing 100k+ spans</a>, we spoke about our <strong>observability</strong> journey from a metrics standpoint to adopting <strong>distributed tracing </strong>as a first-class citizen. This article focuses on the genesis for tracing and our journey into building a <strong>robust distributed tracing stack</strong>.</p><p>Traditional APM (Application Performance Monitoring) tools typically start seeing challenges with the microservices journey. Systems(with microservices) get deeper and things start breaking as the depth of microservices increases. In this article, we will try to cover our story of moving from APM-based monitoring into distributed tracing. This is a multi-series article, where this article will focus on the journey and the reasons for building an in-house distributed tracing platform. Along the way, we will also focus on how tracing has evolved into a first-class citizen, around improving our monitoring.</p><h3>Before You Begin</h3><p>Here are a few rules to follow if you are planning to build your observability stack.</p><p><strong>Rule #1:</strong> Don’t build your observability stack unless you need it.</p><p><strong>Rule #2:</strong> Check the inexpensive options (tracing, metrics, and logs). Explore then exploit.</p><p><strong>Rule #3</strong>: Go Step-by-Step</p><p>1. Structure Your logs: Work at leveraging and scaling a good logging platform.</p><p>2. Metrics — Prometheus + Grafana: This should handle many of the use cases.</p><p>3. Tracing: Use if you have a reasonable number of microservices (or interconnected services)</p><p><strong>Rule #4</strong>: <em>More than 30+ services or about 200+ engineers? </em>Time to evaluate the TCO (Total Cost of Ownership) of <strong>building</strong> an observability platform Vs. <strong>Buy</strong>. Consider factors like Infrastructure Cost, Development Cost, Resource Cost. Software and technologies keep changing, increasing the maintenance cost.</p><p><strong>Rule #5</strong>: Scale -&gt; Tune -&gt; Repeat</p><p><strong>Rule #6</strong>: <em>Not sure yet? </em>Go back to Rule #1 above!</p><h3>Once Upon a Time, We Used APM</h3><p>Somewhere in 2017, as part of observing our systems better, we invested in an APM product. We had few services back then, primarily written using <a href="https://laravel.com/">Laravel</a>. The premise was simple: <em>“Just add this one agent, and you don’t have to worry about your application monitoring”</em>.</p><p>The APM solution worked well even when we moved from the traditional node-based workloads to Kubernetes.</p><p>We were able to monitor a few valuable metrics like:</p><ul><li><strong>General application metrics</strong> — API Level, Pod / Node Level</li><li><strong>Slow Query Monitoring</strong> of all connected data sources — Mysql / Postgres/ Redis etc</li><li><strong>Service boundary deviations</strong> — External Network Calls</li><li><strong>AWS-specific Monitoring</strong> — SQS/ SNS etc</li></ul><p>The APM folks made their solution cloud-native and introduced interesting features like baseline deviation via anomaly detection systems. We also integrated our processes with these solutions and built integrations with Slack, PagerDuty, and leveraged their general ping monitoring for specific cases. Everything was working fine until…</p><h4>Microservices Evolution</h4><p>By late 2017, we embarked on our <strong>microservices</strong> journey and replaced our stack with <em>Golang</em>, as a primary backend language. It changed the entire ballgame.</p><ul><li><strong>No more magic</strong>: Golang is a compiled language. The “Drop the agent here” formula didn’t work with it.</li><li><strong>Change in behavior: </strong>Many engineers started leveraging our metrics system for new services more than APM monitoring due to limited functional capabilities and complex integrations. A large part of incident debugging was conducted on the metrics platform with minimal usage of the APM. APM wasn’t even getting leveraged during real incident scenarios or any kind of internal operability reviews. The usage no longer justified its cost.</li><li><strong>Symptoms</strong>: As the number of services increased, we faced challenges in debugging issues due to service dependencies and high failure points. These symptoms are reflected in our MTTD and other OPEX metrics.</li></ul><h3>The Whys, Whats, and Hows</h3><p>Several questions were haunting us.</p><h4><strong>Engineering</strong></h4><ul><li>Dip due to downstream HTTP/network service being down</li><li>Dip due to query/cache performance</li><li>Dip due to AWS components</li><li>Dip due to surge in traffic</li><li>Dip due to memory / CPU or other infrastructure pieces</li><li>Dip due to a new deployment</li></ul><h4><strong>Business</strong></h4><ul><li>Our metrics infrastructure could not answer high cardinality questions. For example, which merchant is currently seeing issues on a specific route?</li><li>What is the impact of a specific feature going live?</li><li>What is the impact of a third-party system’s erratic behavior?</li></ul><h4><strong>Operations</strong></h4><ul><li>What is a service’s SLI / SLO? How do I define them? What is its baseline?</li><li>How are we going overall from a service, flow, product, app perspective?</li><li>When did something change? Was it sudden or gradual?</li><li>Is there a way to isolate the problem to a specific surface area (reduce MTTD)?</li><li>Did something magical happen? Did something abruptly change and then disappear?</li></ul><h4><strong>Cultural</strong></h4><ul><li>How do we improve the debugging knowledge of the engineers?</li><li>How do we standardize the debugging methods?</li></ul><h4>In Search for Better Observability</h4><p>It was evident that APM was not able to provide answers to our questions. We started looking at the current state of observability and found the following missing pieces:</p><ul><li><strong>Systems getting deeper:</strong> As microservices scaled, it became more challenging to monitor the services and associated dependencies. Especially, the Upstream vs Downstream event collation and correlation became a time-consuming step. The issue was compounded in case of a bad deployment or an incident of any kind.</li><li><strong>Micro Services != Collection of Monoliths:</strong> The traditional APM monitoring failed to answer network/<a href="https://www.sciencedirect.com/topics/computer-science/cascading-effect"> cascading effect</a> problems.</li><li><strong>Pricing</strong>: APM was becoming an increasingly ineffective and expensive solution.</li><li><strong>Dependency Visualization: </strong>There was no clean way to model and visualize the end-to-end visibility in the dependent services. Understanding failure points in a connected graph and correlating them to failure points was becoming painful.</li><li><strong>Finding Hotspots</strong>: Multiple dashboards are required to correlate failure points. We had tons of dashboards on Grafana and our logging platform. Context switches during an incident led to increased cognitive load on developers.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eFD5zeCPGdDChx8Zj4lIZA.png" /></figure><ul><li><strong>Understanding Slow Queries: </strong>For understanding DB chocks and slow queries, the teams were dependent on the DBAs or the infrastructure teams. In certain cases, teams were building specialized pipelines to understand slow database queries.</li><li><strong>Log Fatigue: </strong>An issue led to the skimming of chunks of log data. Logs, being sequential, were adding to debugging time in finding the root cause. The developers had to understand application dependencies to navigate through the logs.</li></ul><p>In a nutshell, the APM system was no more an effective solution to our monitoring requirements. Was <a href="https://en.wikipedia.org/wiki/Telemetry">Telemetry</a> a better approach?</p><h3><strong>Towards an Open Format: Telemetry</strong></h3><p>In recent times, <a href="https://research.google/pubs/pub36356/">google’s dapper</a> and evolution in the observability space has given rise to an open portable format of exposing telemetry data: OpenCensus, OpenTracing, and Open Telemetry. At the core of all of these formats is “<a href="https://newrelic.com/platform/telemetry-data-101">a trace</a>”. In one of our earlier <a href="https://engineering.razorpay.com/tracing-100k-spans-with-razorpay-opencensus-php-fork-d217aa9eca6">articles</a>, we have detailed some interesting aspects and definitions around tracing.</p><h4><strong>Mental Model of a Trace</strong></h4><ul><li><strong>Trace</strong> represents an end-to-end request. It is made up of single or multiple spans.</li><li><strong>Span</strong> is a named, timed operation that represents a piece of a workflow.</li><li><strong>Tags</strong> are key-value pairs that provide details about a span.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GDzarjJ7nPNKHaXMHdAZSg.png" /></figure><h4><strong>Instrumentation Challenges</strong></h4><p>With the above picture in mind, we started building instrumentation around tracing. For Golang, we had to build custom instrumentation. We instead chose to build on an open format, thereby avoiding lock-in with a specific technology or company. However, on the PHP side (a part of our stack on laravel), the available <em>opencensus</em> PHP library had many limitations and open issues. With time, we built a good internal SDK around tracing integration for all Golang apps. On the <em>opencensus</em> side, we forked and made a lot of modifications. Currently, we have the <a href="https://github.com/razorpay/opencensus-php">best PHP opencensus fork</a> available. Read more about <em>opencensus</em> PHP issues and fixes in this <a href="https://engineering.razorpay.com/tracing-100k-spans-with-razorpay-opencensus-php-fork-d217aa9eca6">article</a>.</p><h3><strong>Finding the Best Fit</strong></h3><p>We evaluated 8–10 best PAAS solutions in the industry. These solutions were expensive and only partially met our requirements. For instance, the following are a few points where these solutions fell short:</p><h4><strong>Limited query capabilities for historical traces</strong></h4><p>Popular SaaS vendors support limited querying capabilities on historical traces. Razorpay, being a fintech company, must replay the historical traces for better debugging and compliance purposes.</p><h4><strong>Manage sampling infrastructure in-house</strong></h4><p>With a few vendors, we were required to deploy a sampling component on-premise, which collected 100% of unsampled traces. These components send traces to SaaS vendors. We were looking for an end-to-end managed tracing platform.</p><h4><strong>Pricing</strong></h4><p><strong>Complicated (and obscure) pricing</strong> based on the number of events and services and storage</p><h3><strong>A Step Towards Open Source</strong></h3><h4>Initial Cost Estimations</h4><p>To understand the cost implications of a self-hosting solution, we calculated the cost of storage, operational cost, and engineering cost around extensibility and future scope of work.</p><h4><strong>Storage Cost</strong></h4><p>As per our initial cost estimation with some of our critical services, we needed around 8–10TB of storage for spans and traces (unsampled).</p><p><strong>Calculation: </strong>Span Volume Per Day = # requests per day * spans per trace * bytes per span ~ 8TB unsampled</p><p>One of the ways to reduce the above-mentioned storage requirements is to do some level of trace sampling. For instance:</p><p><strong>Sampling Criteria</strong></p><ul><li>Number of 2xx response codes (e.g. assume this is 95 %)</li><li>Number of 3xx</li><li>Number of 4xx/5xx</li></ul><p>If we sample 2xx at a 1 in 20 rates (5%) and send everything else as it is, we get a sampling rate as (95/20)+5 = 9.6%. Considering this, we would want to send events with low latency and others, keeping the sampling rate at 10%. With this kind of sampling, our estimation was around 1 TB/day of span/trace storage(Assuming, we are sampling around 2xx/3xx responses and sending all 4xx and above unsampled).</p><p>With a 10% sampling, the storage requirements are around 30TB of disk per month. Long-term retention will pose further challenges.</p><p><em>In one of our next set of articles, we will explicitly talk about our sampling strategy.</em></p><h4><strong>Engineering and Operational Cost</strong></h4><p>We wanted to ensure that we are picking up a stack which is:</p><ol><li>Simple and easy to operate</li><li>Can expose monitoring metrics within our current <a href="https://engineering.razorpay.com/scaling-to-trillions-of-metric-data-points-f569a5b654f2">metrics stack</a></li><li>Is container-native.</li><li>Can be extended and able to build custom enrichment and other integration capabilities.</li><li>Doesn’t take a lot of time around engineering maintenance</li><li>Always available as open-source</li><li>Should work with current and future integrations, even if we decide to move the stack to a PaaS offering shortly (Vendor Neutral)</li></ol><h4>Available Alternatives</h4><p>In the initial stages, we wanted to check how tracing works and hence wanted to have some quick wins. We evaluated a few open-source models and came across the following solutions:</p><ol><li><a href="https://skywalking.apache.org/">Apache Skywalking</a></li><li><a href="https://www.jaegertracing.io/">Jaeger</a></li><li><a href="https://expediadotcom.github.io/haystack/">Expedia’s haystack</a></li></ol><p>But, the above-mentioned tools had a few <strong>limitations</strong> too (Evaluations done around early 2020).</p><ol><li>Skywalking was more suitable for Java applications. Our primary backend stack is Golang. Many of its features weren’t helpful to us.</li><li>Adding custom enrichment was challenging in all the above stacks.</li><li>Both Skywalking and Haystack have a heavy dependency on elastic search, which increases cost. Our estimated cost for running this amounts to around 46K USD/ year.</li><li>Haystack has an extremely complicated stack, which means severe maintenance issues.</li><li>Haystack and Skywalking offered limited community support. One was proprietary (Expedia). We found many issues during our initial exploration.</li><li>Cassandra was a huge bottleneck. This was true for Haystack and Jaeger.</li></ol><h3>Finally, what we settled for…</h3><p>After doing many POCs, we started our <strong>in-house distributed tracing journey with </strong><a href="https://www.jaegertracing.io/"><strong>Jaeger</strong></a><strong>. </strong>In the initial days, Jaeger offered all that we wanted from our tracing solution. But, as we went deeper into its adoption, we were struck by a few of its limitations:</p><ul><li><strong>Lack of visibility into derived trace metrics: </strong>RED metrics like p99 latency, call count and the error rate can be derived from aggregated traces. Jaeger focuses on visualizing trace waterfall view and doesn’t give insights on metrics.</li><li><strong>Limited visualization on the dependency graph: </strong>Jaeger’s dependency graph shows dependencies as DAG with calls count as edges. Dependency can be powerful in debugging on visualizing the calls count, error rate, p99 latency between the dependencies.</li><li><strong>No entity-based drill-downs: </strong>There is no way to investigate traces or spans based on entities like API, incoming or outgoing calls, backends, and databases. Every debugging persona starts with a service, an API, or a database.</li></ul><p>In our next article, we will talk about how we overcame many of the limitations that Jaeger had with an open-source system called <a href="https://www.hypertrace.org/"><strong>Hypertrace</strong></a>, our contributions, and our journey into building a robust alternative tool.</p><p>A special thanks to <a href="https://www.linkedin.com/in/annmathew9/">Ann</a> for helping us with the design elements in this post.</p><h3>We are hiring!</h3><p>If you love working on exciting challenges, we are actively hiring for our Engineering team. We are a bunch of ignited minds simplifying financial infrastructure for businesses, and we’re always looking out for great folks. We are looking for both Engineering Managers and Senior Engineers to break barriers and build for the future. <a href="https://razorpay.com/jobs/?utm_src=medium&amp;utm_medium=blog">Apply now.</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a3c72f52d77f" width="1" height="1" alt=""><hr><p><a href="https://engineering.razorpay.com/tracing-our-observability-journey-a3c72f52d77f">Tracing our Observability Journey</a> was originally published in <a href="https://engineering.razorpay.com/">Razorpay Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Highway]]></title>
            <link>https://medium.com/@subinsundar/highway-c9a9ebe85cb0?source=rss-c85f92e3fa0e------2</link>
            <guid isPermaLink="false">https://medium.com/p/c9a9ebe85cb0</guid>
            <category><![CDATA[movies]]></category>
            <category><![CDATA[true-story]]></category>
            <category><![CDATA[child-abuse]]></category>
            <dc:creator><![CDATA[Subin T P]]></dc:creator>
            <pubDate>Wed, 11 May 2016 08:09:25 GMT</pubDate>
            <atom:updated>2016-05-11T08:09:25.205Z</atom:updated>
            <content:encoded><![CDATA[<p>There is this friend of mine who was crazy about movies. Absolutely crazy. She knew the dialogues of all famous movies and it’s technicians’ names by heart. I thought I was the only one mad about movies, but she proved me wrong with her sheer knowledge and passion for movies.</p><p>We used to discuss movies all the time. It was like a breath of fresh air for me. Never have I been so happy to talk to someone. We shared our interpretation of movies. Reviews, technicalities. Everything. We had never met in person but our love for movies united us. It was glorious.</p><p>But somewhere along the line, I realised our taste in movies are completely different. She loved a different set of genres and I loved something else. I haven’t seen even half the number of movies she has. Nevertheless, we still kept talking.</p><p>One day, I asked her — “What is your favourite movie?”.“Now don’t give me that “it’s a long list” kind of diplomatic answer, just name one”, I added.</p><p>“Highway”, came the reply on Whatsapp. I was taken aback. Highway was one of my favourite movies too. But I found it strange. Given her taste in movies, Highway was never her kind of movie. In fact, if I were to take an intersection of movies both of us liked, Highway would probably be the only one in the list.</p><p>Her answer got me curious. “Highway? You serious? I never thought it’s your kind of movie”, I said jokingly.</p><p>There was no response. “Why do you like it so much?”, I asked.</p><p>Whatsapp showed blue ticks next to that message but she never replied.</p><p>In the following days, between our discussions I used to bring up this topic and she would give me some reason or the other to leave the conversation. She was evading the question, on purpose.</p><p>Days passed and I never asked the question again. We continued our conversations like normal.</p><p>One day, out of the blue, I got a message from her around midnight.</p><p>“Subin, do you really want to know why Highway is my favourite movie?”</p><p>Ever so curious, I instantly replied, “Yes”.</p><p>But I was not ready for what she was about to say.</p><p>“Do you remember the last scenes of Highway, after she is brought back home — her uncle visits their house and she tells her family how the very same person had abused her when she was a child?”</p><p>“Yes I do”, I typed hesitantly. I was already getting a sense of what was about to come.</p><p>“I was abused the same way, Subin. By my uncle. When I was in 4th standard”</p><p>I felt numb. I did not know how to react. Should I say sorry? Should I keep silent? I felt helpless.</p><p>“I haven’t been able to sleep properly from that day”, she continued. “I get nightmares sometimes. My life has never been normal since. But I have never said this to anyone. Not even my family. I can’t. What would they feel about me? Will they disown me? Will they be ashamed to have me as their daughter? I don’t know. I don’t want to know. I will keep all this to myself. And now, you know too. Thank you for listening, Subin. I feel a little lighthearted now”</p><p>My blood was boiling. “Shall we kill that bastard?”, I asked instantly. It was probably the old “KSU spirit” in me speaking, but at that moment, I swear I really meant it.</p><p>“He is history now. Passed away couple of years back”, came the reply.</p><p>I breathed a sigh of relief. At least he won’t be able to hurt her now, I said to myself.</p><p>“Shall I kidnap you then? Just like the movie? We could travel a lot”, I asked half-jokingly, hoping to cheer her up.</p><p>“Will you? Please?”, I could imagine her eyes light up as she typed it. She wasn’t joking.</p><p>“I would love to go on a trip like that. Escape from this place. Explore and see the world. Know people, learn their cultures. I just wanna be away from here. I want to sleep peacefully, at least for a few days. This place will only give me nightmares”</p><p>I couldn’t reply. “Will you take me, Subin?”, she asked after a while of silence.</p><p>“One day I will. But I can’t be like Mahabir, you know?”.</p><p>“Haha”. She just laughed. There were no more messages.</p><p>She probably dozed off. But I couldn’t.</p><p>Her experience had completely shaken me. It hurt me, but it equally amazed me.</p><p>I was amazed that in a state which has 100% literacy rate and is known for it’s “culture”, people can do such ruthless acts. I was amazed that she could not tell about this to anyone, not even her father or mother. I was amazed that the bastard was living free all these years even after committing such a heinous crime.</p><p>And this wasn’t the first such story I was hearing. 3 people had shared similar experiences with me in the past. And only one thing was similar among all their stories — They never told this to anyone, fearing what would happen. They feared their family will disown them, or even worse, commit suicide.</p><p>If you are a parent reading this, please understand that sexual abuse is real. It can happen at your place, even if you are in a joint family. Please provide your children the space and the confidence to open up. Please let them know that no matter what happens, you will always be by their side. All this can go a long way in shaping up their future.</p><p>*******************************************************************</p><p>A wise man once said “Art should comfort the disturbed and disturb the comfortable.”</p><p>That is why Highway became her favourite movie. Because she could relate, she could feel Veera’s pain.</p><p>“Highway”, what a perfect name for a movie! The highway through which she entered my heart, and will probably never leave.</p><p>One day, I am gonna take her on that trip I promised. Probably to Himachal. I can’t be like Mahabir for sure, but atleast I can find solace in the fact that I tried.</p><p>When she’s on that hilltop in Himachal, adoring the beauty around and embracing the calmness of the mountains, I will tell her how she’s the bravest woman I’ve ever seen.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c9a9ebe85cb0" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>