<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Venkat Vaidhyanathan on Medium]]></title>
        <description><![CDATA[Stories by Venkat Vaidhyanathan on Medium]]></description>
        <link>https://medium.com/@venkatv?source=rss-e9b3bbbc82dd------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*8TIzDnkkBzEGXkLP.</url>
            <title>Stories by Venkat Vaidhyanathan on Medium</title>
            <link>https://medium.com/@venkatv?source=rss-e9b3bbbc82dd------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 02:36:29 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@venkatv/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Scaling to trillions of metric data points]]></title>
            <link>https://engineering.razorpay.com/scaling-to-trillions-of-metric-data-points-f569a5b654f2?source=rss-e9b3bbbc82dd------2</link>
            <guid isPermaLink="false">https://medium.com/p/f569a5b654f2</guid>
            <category><![CDATA[metrics]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[infrastructure]]></category>
            <category><![CDATA[prometheus]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Venkat Vaidhyanathan]]></dc:creator>
            <pubDate>Fri, 30 Jul 2021 12:39:11 GMT</pubDate>
            <atom:updated>2021-07-30T12:39:11.400Z</atom:updated>
            <content:encoded><![CDATA[<p>Authors: <a href="https://medium.com/u/e9b3bbbc82dd">Venkat Vaidhyanathan</a> and <a href="https://medium.com/u/4d5cfcb0ba06">Vaibhav Khurana</a></p><p>As a fintech organization, we move billions of dollars every month. Our customers and merchants have entrusted us with a paramount responsibility. To handle our ever-growing business, building a robust observability stack is not just “nice to have”, but absolutely essential. And all of this starts with better monitoring and metrics.</p><p>Over the past few years, Prometheus has become the de-facto platform for metric collection. Razorpay has perhaps been one of the first companies in the country to have gone with a production-grade Kubernetes infrastructure (somewhere late 2016). We wanted to ensure that applications and associated infrastructure should have very little friction to emit and consume metrics. As we have grown in traffic, we had to revisit some of our fundamental assumptions and ways for improving our metrics and observability platform. This article tries to lay out some of the challenges, our journey into building a robust metric collection and alerting platform and also provides insights into how we are constantly improving the same.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*cMdpFUS6ckbGEz4F" /></figure><h3>Humble Beginnings: 2016- 2018</h3><p>There are a bunch of older technologies like Nagios and Icinga which have traditionally been used to monitor applications and infrastructure. Unfortunately, none of these technologies are container aware. We also had a strong requirement of running the monitoring infrastructure in-house. The dynamic nature of Kubernetes demands monitoring systems to detect the changing infrastructure. By 2018, we had close to 30+ services in production on AWS.</p><p>During the initial days of our Kubernetes journey, we had a fairly standard setup of Prometheus as depicted below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*IYHRFQc83i_dVL2P" /><figcaption>Prometheus Architecture</figcaption></figure><p>Broad Categories of metrics/exporters used: :</p><ol><li>Infrastructure metrics(CPU, Memory, Disk, etc,): Captured via Node Exporter daemon on each node.</li><li>Kubernetes Metrics: The health of the Kubernetes clusters are captured from sources like cAdvisor, kubelet and kube-state-metrics.</li><li>AWS Metrics(RDS, SQS, etc): Exported to Prometheus via Cloudwatch exporter.</li><li>Application Metrics: Our applications emit business metrics using standard Prometheus libraries, which are scraped and ingested into Prometheus</li><li>Application Metrics(Legacy): Some legacy applications use statsd, which internally emit Prometheus metrics, which are again ingested and scrapped into Prometheus.</li></ol><p>Prometheus itself was running as a statefulset(inside Kubernetes) with a disk backed setup(statefulset with EBS backed PVCs). Grafana and alert manager were configured to read from Prometheus. While Grafana provides the visualization, AlertManager is configured to send alerts to various channels like slack, pager duty, etc. While all of this worked well, at scale, we encountered the following issues:</p><ol><li>Prometheus inherently isn’t HA(Highly Available). So, it was always a single point of failure.</li><li>As the metrics grew in numbers, visualizing metrics over a longer period of time was proving to be even more disastrous.</li><li>The only way to scale up Prometheus was doing a vertical scale-up. This led to increased costs on the monitoring infrastructure.</li><li>Read/Write Splits: Because of the Prometheus&#39;s inherent design, the writes and reads were being served off a single Prometheus instance.</li><li>Besides, we were also getting hit by a bunch of performance issues like the below:</li></ol><ul><li><a href="https://github.com/prometheus/prometheus/issues/8533">https://github.com/prometheus/prometheus/issues/8533</a> — Kernel out of memory</li><li><a href="https://github.com/prometheus/prometheus/issues/8392">https://github.com/prometheus/prometheus/issues/8392</a> — High CPU Usage</li><li><a href="https://github.com/prometheus/prometheus/issues/8310">https://github.com/prometheus/prometheus/issues/8310</a> — Higher memory usage after upgrades.</li></ul><p>Some of the above reduced our overall confidence in upgrading Prometheus itself. This was to be done carefully, monitored and then rolled out. At the end of 2018, we were looking to answer the following:</p><ol><li>How do we scale Prometheus with the increasing workload?</li><li>How do we split between Read vs Write workloads?</li><li>How do we retain metrics for a longer period and still be able to query faster?</li><li>How do we monitor Prometheus itself?</li></ol><h3>Next Chapter: 2018–2020</h3><p>During this period, we were moving from a single Kubernetes cluster to multiple Kubernetes clusters (for achieving High Availability) and the number of microservices on the setup has crossed a 100 mark by this time further increasing the impact radius. Now because Prometheus runs inside Kubernetes, we have multiple Prometheus deployments, each one running respectively in their own clusters. How do we make Grafana query and visualize across these multiple Prometheus setups? During this time, <a href="https://github.com/thanos-io/thanos">Thanos</a> was shining like a bright star.</p><p><em>“Thanos is a set of components that can be composed into a highly available metric system with unlimited storage capacity, which can be added seamlessly on top of existing Prometheus deployments.”</em></p><p>We went ahead with deploying Thanos and complemented it with <a href="https://github.com/trickstercache/trickster">Trickster</a> as caching layer for optimising the queries from Grafana.</p><p>The overall architecture looked like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/864/0*MnBzJJgcaz4u9FTY" /><figcaption>Thanos — High-Level Architecture</figcaption></figure><p>The above architecture solved a bunch of problems listed above:</p><ul><li>HA mode for running Prometheus.</li><li>Long term metric retention was solved via Thanos, as Thanos writes to remote deep storage like S3.</li><li>Improved query performance.</li><li>Distributed reads on multiple Prometheus servers via Thanos Querier.</li></ul><p>While the above worked great for a long period of time, this led to the following issues:</p><ol><li>Thanos remote write to S3 causes the following issues:</li><li>Most fetches have to happen from the remote storage and then be computed in memory</li><li>The remote store is a flat directory. So, for all large time range queries, it involves an S3 list operation and then a fetch.</li><li>Thanos writes the time series data every few hours and this is an extremely network and cost-intensive operation.</li><li>In addition, there are situations where the Grafana dashboards do not load very fast, leading to query timeouts. This goes back to the remote fetch operation on S3 and OOM situations on Thanos Store.</li><li>Our infrastructure cost was almost around $12k/month making it extremely cost-prohibitive.</li><li>For improved reliability, we had Prometheus HA pairs, where Thanos does the deduplication. This almost doubles the cost.</li><li>More importantly: Our metric cardinality was very high. To present an example, let’s say we want to query the number of successful HTTP requests over a period of time, the query would look like the following:</li></ol><pre>sum(rate(payment_create_http_requests[2m]))</pre><p>Now, add multiple additional dimensions to this:</p><ol><li>Payment Method: Card vs Netbanking</li><li>Payment Network (Visa, Mastercard….) or Bank(HDFC, ICICI, etc)</li><li>International vs domestic</li><li>TPV vs non TPV</li><li>Merchant MCC code</li></ol><p>And the list goes on….</p><p>In essence, handling such high cardinality was to be done in memory, after the remote fetch. Multiply this query(with the above dimensions) over a longer duration, and Thanos stopped functioning.</p><p>Overall, there was a lot of babysitting to do, with tons of cost going there.</p><p>To get a sense of where we stood by early 2020, our Prometheus data looked like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/650/0*EQHqGM38JqMUI1q6" /><figcaption>Prometheus Data Growth — 2018–2020</figcaption></figure><h3>The final stroke: 2020 and Beyond</h3><p>At the beginning of 2020, we started searching for various solutions. We went from existing open-source ones to proprietary formats to others. The basic questions that we had were the following:</p><ol><li>Can we split the read and write workloads on Prometheus?</li><li>Can we reduce the cost? Can we run a large portion of the workloads on spot instances?</li><li>Can we do better long term retention?</li><li>Can we handle the high dimensional cardinality issue we mentioned above?</li><li>Can we leverage existing open source solutions and not have any specific vendor lock-ins?</li></ol><p>We executed a variety of POCs on various solutions and finally arrived at the following technologies:</p><ol><li><strong>Uber’s M3DB </strong>— We discarded this, primarily because we wanted an open format like Prometheus and also wanted better community support. Plus, we didn’t want to change our metric emission philosophy</li><li><strong>Thanos</strong> — Thanos had itself gone through a lot of optimizations. We weren’t comfortable, as the fundamental design still imposed some of the above challenges. Our POCs didn’t yield successful results either.</li><li><strong>Cortex</strong> — Seemed extremely promising</li><li><strong>Victoria Metrics</strong> — The clear winner.</li></ol><p><em>Note: There were tons of other POCs we did with various vendors and platforms. The above look more noteworthy.</em></p><p>We started our final phase of experiments with Cortex(the project was created by Grafana themselves, claimed impressive numbers, and was being dog-fed into Grafana’s saas platform among others). For a detailed overview of the cortex architecture, please refer <a href="https://cortexmetrics.io/docs/architecture/">here</a>. Very specifically, we ran into the following issues:</p><ol><li><strong>Cost and Availability: </strong>Ingesters store metrics in memory and eventually write to remote storage like S3/Cassandra. At scale, ingesters started going down. This led to availability issues and metric loss. Scaling this up means adding more memory and expanding vertically.</li><li><strong>Support and Maintainability:</strong> While, we would have ideally liked the remote storage to be S3, cortex specifically didn’t have support for <a href="https://docs.aws.amazon.com/AmazonS3/latest/API/sig-v4-authenticating-requests.html">S3’s V4 signing</a> (at the point of this evaluation). This means we were forced to set up a separate Cassandra cluster for handling the remote storage. This posed an extra component to maintain, and overall, maintaining Cassandra’s health, had to be dog-fed to Cortex. We didn’t want to venture with DynamoDB, as we strictly wanted to keep away from vendor lock-ins of any kind. All of this added overhead to the monitoring system.</li><li><strong>Maintainability: </strong>Cortex maintains a cache layer via Memcached/Redis. While we would have preferred Redis (as we had enough operational experience), Redis was termed “experimental” and only Memcache seemed GA. Again, another overhead in the monitoring platform.</li><li><strong>Hardware and Cost: </strong>Our minimal performance testing, led us to the following hardware: 10 ingester pods(nodes), around 5 distributors, 3 consul nodes, 5 Cassandra nodes. Worse, the ingester maintains the state in memory and cannot be run on spot nodes(preemptive). We lose a node and we lose associated metrics for that period.</li><li><strong>Ease: </strong>Overall, the number of moving parts(Cortex itself, Consul, Cassandra, Memcache), maintaining availability across all, was becoming extremely challenging. We didn’t want to make monitoring so complex.</li><li><strong>Functional Guarantees: </strong>And finally some of our high cardinality queries were still failing to load well.</li></ol><p>We started investigating <a href="https://victoriametrics.com/">Victoria Metrics</a> next. Victoria Metrics has a simple architecture as described below. At the outset, this was a breeze (No S3, Cassandra, sidecar or any extra frills).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*E4VWSSxwu2r0serc" /><figcaption>Victoria Metrics — Architecture</figcaption></figure><p>The following are some of the basic observations we derived from Victoria Metrics:</p><ul><li>Simple components, each horizontally scalable.</li><li>Clear separation between writes and reads.</li><li>Runs from default configurations, with no extra frills.</li><li>Default retention starts with 1 month</li><li>Storage, ingestion, and reads can be easily scaled.</li><li>High Compression store ~ 70% more compression.</li><li>Currently running in production with commodity hardware with a good mix of spot instances.</li><li>Successfully ran some of the worst Grafana dashboards/queries that have historically failed to run</li></ul><h3>Migrating to VM</h3><p>We first pointed our Prometheus servers to remote write to Victoria Metrics. We were observing potential issues if any for a few days. In parallel, we started pointing a few problematic Grafana dashboards to VMSelect. We then went ahead to tune some of the VM configs to maintain an optimal setup. Over the next few days, we started moving the underlying VM instances to spot(preemptible) nodes to keep costs in control. Finally, once the infra and setup looked stable, we did an overnight shift of changing the default Prometheus (Thanos) query endpoints to VM on Grafana. The same was done with Alert Manager too. We were able to finish the entire migration in under a month across 4 different Kubernetes clusters.</p><p><strong>Complement Alert Manager with vmAlert:</strong> vmAlert evaluates alert rules against victoria metrics and sends firing alerts to the configured alert managers which in turn notified the relevant teams via Slack, Pager etc. Alerts can then be silenced(when needed) and allowed better handling of alerts.</p><h3>Next Steps</h3><p>With the above victory, we went to do the following:</p><p><strong>Replace Prometheus with vmAgent: </strong>Prometheus is resource-heavy. In addition, when we were running Prometheus with Thanos, we also encountered a whole lot of <a href="https://en.wikipedia.org/wiki/Write-ahead_logging">WAL (Write- Ahead Log) </a>corruption issues. This was bundled with delays in pushing metrics to a remote VM store. We instead replaced Prometheus itself with VMAgent. This gave us the following advantages:</p><ol><li>VMAgents are lightweight.</li><li>They are Prometheus compatible and blends well with the VM servers</li><li>Uses lesser storage and is stateless.</li><li>The above setup can be run in HA mode.</li><li>Using spot(preemptible) instances, we can achieve significant cost optimization.</li></ol><p><strong>Monitor Victoria Metrics: </strong>Who monitors the monitor? Well, we have an additional operational Kubernetes cluster for running our observability and other regular DevOps stack. We set up a smaller VM Cluster that internally monitors our primary victoria metrics cluster. In essence, we monitor VM via VM.</p><h3>What’s the cost?</h3><ul><li>Only vmStorage runs on On-Demand Instances</li><li>vmSelect, vminsert and vmalert runs on 100% spot nodes with Horizontal Pod Autoscaler enabled</li><li>5 vmStorage nodes: ~$13</li><li>3 spot nodes for other server components, similar to r5a.2xlarge= ~$3</li><li>2vm agent pods/division group with requirements as 500M and 10mi=~$2</li></ul><p><strong>Total Spends ~ $18/day (Excluding the cost of EBS storage and data transfer)</strong></p><h3>And how are we scaling, you ask?</h3><p>We collected the below metrics after almost 8 months of battle testing victoria metrics in production. By this time, as we notice, our metric ingestion rate had increased significantly. We are able to retain metrics for over a month. With the below numbers and the above cost($18), this looks impressive. Right?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/756/0*vXlQHHm8kaZzQX8j" /><figcaption>Victoria Metrics — Metric Growth</figcaption></figure><h3>Road Ahead</h3><p>While most of the above things work, we periodically do see extreme cardinality issues. This usually happens with third party open source software that doesn’t use Prometheus labels or comply with best practices. So, being able to detect these, is still one big challenge. We are exploring a variety of ways to prevent these metrics from being ingested or somehow rate limit these kinds of metrics into the system.</p><p>On a closing note, while our journey on observability has reached a good stage of maturity, we are still learning new aspects of our platform every other day. This, we believe, will only strengthen our work in making our applications and systems far more robust. We are doing some fairly interesting work on alert fatigue reduction, workload forecasting and building an extremely deep observability stack.</p><p>Finally, we are also building capacity to increase the retention of our metrics. At this point, we are storing a month old data. While we are expanding this to 3 months on the victoria metrics side, we are also planning to export these into our data lake for longer-term work around data science and workload behaviour modelling.</p><h3>Learnings</h3><p>During this journey, we learnt tons of things. The following are some salient aspects we wanted to capture:</p><ol><li><strong>Keep the house clean:</strong> Don’t complicate the infrastructure. Keep it simple and moving and be ready to change, as the need arises. Always be open to change.</li><li><strong>Stick to open standards:</strong> Try as much as possible to avoid vendor lock-ins. This will help move things around various container and cloud strategies. This will also expose you to a thriving community of the open-source ecosystem.</li><li><strong>Don’t go by marketing material:</strong> Try the solution to see if it fits your needs. Every problem is unique and every technology comes with its own pros and cons. This includes making a good choice between buy vs build decisions as well. Some of the technologies failed for us, but could potentially work for others.</li><li><strong>Strive for regular operational reviews:</strong> Observability is a lifelong journey. Have regular reviews with the development team, start working to reduce <a href="https://www.crazyegg.com/blog/glossary/what-is-a-vanity-metric/">vanity metrics</a> and ensure the right cardinality measures. This will make your monitoring sound like good poetry.</li></ol><h3>We are hiring</h3><p>If the work we do excites you, we are actively hiring for our engineering team. We are a bunch of ignited minds simplifying financial infrastructure for businesses. We are always looking out for great folks. Apply via our <a href="https://razorpay.com/jobs/?utm_src=medium&amp;utm_medium=blog">jobs page</a> or reach out to me at venkat.v@razorpay.com</p><h3>References</h3><ol><li><a href="https://github.com/thanos-io/thanos">https://github.com/thanos-io/thanos</a></li><li><a href="https://github.com/tricksterproxy/trickster">https://github.com/tricksterproxy/trickster</a></li><li><a href="https://cortexmetrics.io/docs/architecture/">https://cortexmetrics.io/docs/architecture/</a></li><li><a href="https://docs.aws.amazon.com/AmazonS3/latest/API/sig-v4-authenticating-requests.html">https://docs.aws.amazon.com/AmazonS3/latest/API/sig-v4-authenticating-requests.html</a></li><li><a href="https://victoriametrics.com/">https://victoriametrics.com/</a></li><li><a href="https://prometheus.io/">https://prometheus.io/</a></li><li><a href="https://www.meetup.com/Bangalore-Monitoring-and-Observability-Meetup/events/273408763/">https://www.meetup.com/Bangalore-Monitoring-and-Observability-Meetup/events/273408763/</a></li><li><a href="https://www.youtube.com/watch?v=kxWK737DPUQ">https://www.youtube.com/watch?v=kxWK737DPUQ</a></li><li><a href="https://en.wikipedia.org/wiki/Write-ahead_logging">https://en.wikipedia.org/wiki/Write-ahead_logging</a></li><li><a href="https://www.crazyegg.com/blog/glossary/what-is-a-vanity-metric/">https://www.crazyegg.com/blog/glossary/what-is-a-vanity-metric/</a></li><li><a href="https://sflanders.net/2018/12/18/an-intro-to-prometheus/">https://sflanders.net/2018/12/18/an-intro-to-prometheus/</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f569a5b654f2" width="1" height="1" alt=""><hr><p><a href="https://engineering.razorpay.com/scaling-to-trillions-of-metric-data-points-f569a5b654f2">Scaling to trillions of metric data points</a> was originally published in <a href="https://engineering.razorpay.com/">Razorpay Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Letter from my Past — A letter from my 20 year old self]]></title>
            <link>https://medium.com/@venkatv/a-letter-from-my-past-a-letter-from-my-20-year-old-self-696c70413da1?source=rss-e9b3bbbc82dd------2</link>
            <guid isPermaLink="false">https://medium.com/p/696c70413da1</guid>
            <dc:creator><![CDATA[Venkat Vaidhyanathan]]></dc:creator>
            <pubDate>Mon, 24 Feb 2020 19:44:27 GMT</pubDate>
            <atom:updated>2020-02-24T19:44:27.390Z</atom:updated>
            <content:encoded><![CDATA[<h3>A Letter from my Past — A letter from my 20 year old self</h3><p><em>Disclaimer: I am migrating my current personal(non tech) blog from blogger to medium. Unfortunately, import options in medium do not manage publishing date well and hence might appear as newer content. Originally posted on </em>SEPTEMBER 13, 2010<em>. End of Disclaimer.</em></p><p>I’ve traveled in life this far, with some advice. Unfortunately, I can only time travel in one direction: forward. I can’t go back and tell myself: “You should have gone out of this country and made a few more bucks than being here”.</p><p>But perhaps some of my advice will be useful to people who were where I was then: 20 yrs.</p><p><strong>Save, Save and Save More: </strong>I’ve always had good jobs, but I haven’t always spent what I got wisely. I’m not saying that I should have been a penny pincher, exactly. But there’s a big difference between buying everything on a whim, and buying nothing.</p><p>I wish I had been a little more conscious of that balance. Catching up later in life is much harder than starting early. Even starting a few years later can have a big impact. A Simple Recurring Deposit of Rs 1L for 20 years(at an interest rate of 7.75%) yields an interest of Rs.155000 + Original. 10 years later, its only Rs. 77500.</p><p>I almost wrote a Rs.10 Cr. (back then) check on my name when I was 20 to invest in my retirement — as a kind of forced savings plan. I was good(or probably scared) at paying off loans quickly, but not so good at just saving money. I’ve regretted not doing that.</p><p><strong>Buy just what’s needed: </strong>In the movie , Tyler says “The stuff you own ends up owning you”. That statement has really resonated with me. On the one hand, everything is saying “buy more”: TV, magazines, the Internet, buses and bus stops even. Everyone wants you to buy their stuff. Getting stuff is often much easier than getting rid of it. “Just throw it away” I hear you saying… I’ve found it’s not as easy as it sounds to throw away a perfectly good — whatever, even if you aren’t using it.</p><p>These days I think hard (and probably still not as hard as I should) before I buy something new. I admire the people who are trying to live with less stuff, even though I’ll probably never be one of them.The more stuff you have, the more stuff you have to keep organised. The more gadgets you have, the more you need to ensure them and the more you need to protect them.</p><p><strong>Follow Your Bliss: </strong>This has really made my life enjoyable(or sometimes miserable). I get paid for doing what I’d be doing anyway. For me, it’s working with computers. I really, truly, love that. I’m convinced that everyone can make a living at doing something they really want to do. Well then you might ask, who is going to clean the garbage outside my house? Funny you should ask. I had a friend back in high school, his father was a professor at the University. During our summer holidays, he used to call the folks in our colony on every sunday morning and we used to clean our community. He set up a collective fund from the colony for this, distributed part of the money to us and kept the remaining to the municipality folks. Figure out what you really like to do, and spend some effort trying to come up with a way to make money out of it. I truly truly believe, that a job you love is worth much more than a job that pays well which you hate/dislike.</p><p><strong>Follow through, till the end: </strong>I love starting new things, and i mean starting. It’s easy and exciting to start something. Then you realize there are a zillion little pieces, all similar, that you have to, sand, glue, realize you did it wrong, and fix…</p><p>That thing that was so great when looked at from a distance can really become tedious when you’re in the trenches doing the work to make it a reality. After a string of half completed projects, it starts to get pretty oppressive…and sometimes leads into severe trauma, that would take years to fix.</p><p>I am right now very very careful about starting new projects, be it my theatre, or my music or anything fun, outside my job. If I’m going to start something new, I have to decide what existing project I’ll give up on. Or can this new, exciting project wait until an existing project is done?</p><p>I really enjoy the satisfaction of having something done, and being able to make use of — whatever I was working on.</p><p><strong>The tool called “Perception”: </strong>For quite a long while, confused with the topics like “start with the end in mind”, in any project, i was only seeing the boring things that stand between me and the completion.Very recently, I have started looking at similar projects and think about how good it will feel to be done with it. This has been instrumental in improving my follow-through.</p><p>And this all comes from (and trust me, I hate this cliche) looking on the bright side. So, don’t concentrate on having to get up early, deal with the traffic and travelling, spend long hours inside an AC room, starting at your 17inch beast, among other things…. Imagine you are on a miracle flight, that could change things. I know, i know its a very very cheap trick. I know, but like placebos(The placebo effect: <a href="http://wn.wikipedia.org/wiki/Placebo),">http://wn.wikipedia.org/wiki/Placebo),</a> its shockingly effective.</p><p><strong>Believe, Believe and Believe: </strong>This is almost my superpower. It’s not so much that I know I can do anything, it’s more the converse; I know, what i don’t know. And you would be surprised to know that i learnt that just a few days back, in a very trivial circumstance. That attitude has made it so that I pretty much <em>can</em> do anything… This is another cheap trick like the above, but it’s one i believe is serving me well now. If you decide you can’t do something, you’re probably going to end up being right.</p><p><strong>Learn to leave your frustrations: </strong>This has really been the hard part for me, all these years. The biggest stumbling block to the above idea in “Belief” for me, also stems from here. One thing, if at all, i have to say has been strong in my life, is the belief in God. One of my friends recently asked me this : “What is that one thing for which you could give up anything else in your life”. I had an answer to that question. And when that one thing is obvious, why get frustrated about that “something” ? Perhaps, that “something”, is not all that important in your life. The band, the WeatherThans <em>(</em><a href="http://www.theweakerthans.org/discography/reconstructionsite_content.html">http://www.theweakerthans.org/discography/reconstructionsite_content.html</a>) have this song where they say “Throw away my misery; It never meant that much to me; It never sent a get-well card.” Just let go of what makes you mad or sad. If you focus on it, you’re just giving it power. Let your frustrations go and give power to the successes. I could go on with this, probably even publish a paper or two on this. For me, its more of an internal healing process.</p><p><strong>And Finally:</strong> Now go forth watch of (http://www.imdb.com/title/tt096928) <em>Bill and Ted</em> who said “Be excellent to one-another.” I genuinely believe in one life with one body and one mind. I cannot borrow or change any of these(life, mind and body). Live, let live and be happy :).</p><p><em>Originally published at </em><a href="http://dsci.blogspot.com/2010/09/letter-from-my-past-letter-from-my-20.html"><em>http://dsci.blogspot.com</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=696c70413da1" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Shall we talk Philospphy?]]></title>
            <link>https://medium.com/@venkatv/shall-we-talk-philospphy-cffa32fad4f?source=rss-e9b3bbbc82dd------2</link>
            <guid isPermaLink="false">https://medium.com/p/cffa32fad4f</guid>
            <dc:creator><![CDATA[Venkat Vaidhyanathan]]></dc:creator>
            <pubDate>Mon, 24 Feb 2020 19:19:10 GMT</pubDate>
            <atom:updated>2020-02-24T19:19:10.764Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Dislaimer:</em> <em>I am migrating my current personal(non tech) blog from blogger to medium. Unfortunately, import options in medium do not manage publishing date well and hence might appear as newer content. Originally posted at the canonical link below on </em>FRIDAY, JULY 18, 2008<em>. End of Disclaimer.</em></p><p>When i was in school, I decided I wanted to study philosophy, at some point. But then, practical aspects of life and money, sort of made it impossible to take up philosophical studies, as a serious subject in this country. I followed the others around my school, went into an engineering college and all that. But what is important here is, I had several motives, some more honorable than others. One of the less honorable was to shock people. You know, Schools and Colleges are always regarded (or atleast i did) as job training where I grew up. So studying philosophy seemed an impressively impractical thing to do. Sort of like slashing holes in your clothes or putting a board pin through your ear, belly, and your tongue which were other forms of impressive impracticality then just coming into fashion.</p><p>But I had some more honest motives as well. I thought studying philosophy would be a shortcut straight to wisdom. You know, all my friends would just end up with some “specific” domain knowledge, based on their coursework. But, I would be learning what was really what.</p><p>I’d tried to read a few philosophy books. Not recent ones; But I tried to read Plato and Aristotle. I doubt I believed I understood them, but they sounded like they were talking about something important. Honestly, I learnt a lot in my Discrete Mathematics class, but I didn’t learn much from these gentlemen. And yet my plan to study philosophy remained intact. It was my fault I hadn’t learned anything. Then, i was suggested Berkley by someone. I did give Berkeley’s <em>Principles of Human Knowledge</em> another shot in college. Anything so admired and so difficult to read must have something in it, if one could only figure out what. I still don’t understand Berkeley, after almost a decade now. I have a nice edition of his collected works. Will I ever read it? Seems unlikely. The difference between then and now is that now I understand why Berkeley is probably not worth trying to understand. I think I see now what went wrong with philosophy, and how we might fix it.</p><p>Philosophy doesn’t really have a subject matter in the way math or history or most other university subjects do. There is no core of knowledge one must master. The closest you come to that is a knowledge of what various individual philosophers have said about different topics over the years. Few were sufficiently correct and the rest, been forgotten in the sands of time or into fights over who discovered what first.</p><p>Heard of Formal Logic? It does seem to me very important to be able to flip ideas around in one’s head: to see when two ideas don’t fully cover the space of possibilities, or when one idea is the same as another but with a couple things changed. But did studying logic teach me the importance of thinking this way, or make me any better at it? I don’t know.</p><p>There are things I know I learned from studying philosophy. The most dramatic I learned immediately is that `I` don’t exist. `I` am (and you are) a collection of cells that lurches around, driven by various forces, and calls itself I. But there’s no central, indivisible thing that your identity goes with. You could conceivably lose half your brain and live. Which means your brain could conceivably be split into two halves and each transplanted into different bodies. Imagine waking up after such an operation. You have to imagine being two people.</p><p>The real lesson here is that the concepts we use in everyday life are fuzzy, and break down if pushed too hard. Even a concept as dear to us as “I” or “Self”. It took me a while to grasp this, but when I did, it was fairly sudden, like someone in the nineteenth century grasping evolution and realising the story of creation they’d been told as a child was all wrong. [1] Outside of math there’s a limit to how far you can push words; in fact, it would not be a bad definition of math to call it the study of terms that have precise meanings. Everyday words are inherently imprecise. They work well enough in everyday life that you don’t notice. Words seem to work, just as Newtonian physics seems to. But you can always make them break if you push them far enough.</p><p>I would say that this has been, unfortunately for philosophy, the central fact of philosophy. Most philosophical debates are not merely afflicted and driven by confusions over words. Do we have free will? Depends what you mean by “free”. Do abstract ideas exist? Depends what you mean by “exist.”</p><p>This brings me to an interesting question. How did things get this way? Can something, people have spent thousands of years studying really be a waste of time? In fact, some of the most interesting questions you can ask about philosophy; The most valuable way to approach the current philosophical tradition may be neither to get lost in pointless speculations like Berkeley, nor to shut them down, but to study it as an example of reason gone wrong.</p><p>Western philosophy really begins with Socrates, Plato, and Aristotle.There started to be a lot more analysis. I suspect Plato and Aristotle were encouraged in this by progress in math. Mathematicians had by then shown that you could figure things out in a much more conclusive way than by making up fine sounding stories about them. [2]<br>People talk so much about abstractions now, that we don’t realise what a leap it must have been when they first started to. It was presumably many thousands of years between when people first started describing things as hot or cold and when someone asked “what is heat?” No doubt it was a very gradual process. Aristotle in particular reminds me of the phenomenon that happens when people discover something new, and are so excited by it that they race through a huge percentage of the newly discovered territory in one lifetime. If so, that’s evidence of how new this kind of thinking was. [3] Something similar happened when people first started trying to talk about abstractions. Much to their surprise, they didn’t arrive at answers they agreed upon. In fact, they rarely seemed to arrive at answers at all.They were in effect arguing about artefacts induced by sampling at too low a resolution.</p><p>Lets just step back to see what really was the goal then. Aristotle’s goal, for instance was to find the most general of general principles. The examples he gives are convincing: an ordinary worker builds things a certain way out of habit; a master craftsman can do more because he grasps the underlying principles. The trend is clear: the more general the knowledge, the more admirable it is. But then he makes a mistake-possibly the most important mistake in the history of philosophy. He has noticed that theoretical knowledge is often acquired for its own sake, out of curiosity, rather than for any practical need. So he proposes there are two kinds of theoretical knowledge: some that’s useful in practical matters and some that isn’t. Since people interested in the latter are interested in it for its own sake, it must be more noble. So he sets as his goal in Metaphysics: the exploration of knowledge that has no practical use. Which means no alarms go off when he takes on grand but vaguely understood questions and ends up getting lost in a sea of words.</p><p>His mistake was to confuse motive and result. Certainly, people who want a deep understanding of something are often driven by curiosity rather than any practical need. But that doesn’t mean what they end up learning is useless. So while ideas don’t have to have immediate practical applications to be interesting, the kinds of things we find interesting will surprisingly often turn out to have practical applications.That’s what makes theoretical knowledge prestigious. It’s also what causes smart people to be curious about certain things and not others; our DNA is not so disinterested as we might think.The reason Aristotle didn’t get anywhere in the Metaphysics was partly that he set off with contradictory aims: to explore the most abstract ideas, guided by the assumption that they were useless.</p><p><strong>So, is there a problem?</strong><br>And since his work became the map used by generations of future explorers, he sent them off in the wrong direction as well. [4] Perhaps worst of all, he protected them from both the criticism of outsiders and the promptings of their own inner compass by establishing the principle that the most noble sort of theoretical knowledge had to be useless.</p><p>Metaphysics is mostly a failed experiment. A few ideas from it turned out to be worth keeping; the bulk of it has had no effect at all. The Metaphysics is among the least read of all famous books. It’s not hard to understand the way Newton’s Principia is, but the way a garbled message is.</p><p>In the intervening years an unfortunate idea took hold: that it was not only acceptable to produce works like the Metaphysics, but that it was a particularly prestigious line of work, done by a class of people called philosophers. No one thought to go back and debug Aristotle’s motivating argument. And so instead of correcting the problem Aristotle discovered by falling into it-that you can easily get lost if you talk too loosely about very abstract ideas-they continued to fall into it.</p><p>If you write in an unclear way about big ideas, you produce something that seems tantalisingly attractive to inexperienced but intellectually ambitious students. Till one knows better, it’s hard to distinguish something that’s hard to understand because the writer was unclear in his own mind from something like a mathematical proof that’s hard to understand because the ideas it represents are hard to understand. To someone who hasn’t learned the difference, traditional philosophy seems extremely attractive: as hard (and therefore impressive) as math, yet broader in scope. That was what lured me in as a high school student. So, what’s even worse is, when things are hard to understand, people who suspect they’re nonsense generally keep quiet. There’s no way to prove a text is meaningless. The closest you can get is to show that the official judges of some class of texts can’t distinguish them from placebos. [5] Because philosophy’s flaws turned away the sort of people who might have corrected them, they tended to be self-perpetuating.</p><p><strong>So, is there a solution? </strong><br>Here’s an intriguing possibility. Perhaps we should do what Aristotle meant to do, instead of what he did. The goal he announces in the Metaphysics seems one worth pursuing: to discover the most general truths. That sounds good. But instead of trying to discover them because they’re useless, let’s try to discover them because they’re useful. Lets use, applicability, as a guide to keep us from wondering off into a swamp of abstractions. Instead of trying to answer the question:</p><p>What are the most general truths?</p><p>let’s try to answer the question</p><p>Of all the useful things we can say, which are the most general?</p><p>The goal is the same as Aristotle’s; we just approach it from a different direction.</p><p>As an example of a useful, general idea, consider that of the controlled experiment. There’s an idea that has turned out to be widely applicable. Some might say it’s part of science, but it’s not part of any specific science; it’s literally meta-physics (in our sense of “meta”). The idea of evolution is another. It turns out to have quite broad applications-for example, in genetic algorithms and even product design. Frankfurt’s distinction between lying and bullshitting seems a promising recent example. [6]</p><p>These seem to me what philosophy should look like: quite general observations that would cause someone who understood them to do something differently.</p><p>One drawback of this approach is that it won’t produce the sort of writing that gets you tenure. And not just because it’s not currently the fashion.Here’s the exciting thing, though. Anyone can do this.</p><p>If it seems like a daunting task to do philosophy, here’s an encouraging thought. Though the first philosophers in the western tradition lived about 2500 years ago, it would be misleading to say the field is 2500 years old, because for most of that time the leading practitioners weren’t doing much more than writing commentaries on Plato or Aristotle while watching over their shoulders for the next invading army. It didn’t shake itself free till a couple hundred years ago, and even then was afflicted by the structural problems I’ve described above. If I say this, some will say it’s a ridiculously overbroad and uncharitable generalization, and others will say it’s old news, but here goes: judging from their works, most philosophers up to the present have been wasting their time. So in a sense the field is still at the first step. [7]</p><p>Hmm… That sounds a preposterous claim to make. Civilization always seems old, because it’s always the oldest it’s ever been. The only way to say whether something is really old or not is by looking at structural evidence, and structurally philosophy is young; it’s still reeling from the unexpected breakdown of words.</p><p>Philosophy is as young now as math was in 1500. There is a lot more to discover.</p><p>[1] It was harder for Darwin’s contemporaries to grasp this than we can easily imagine. The story of creation in the Bible is not just a Judeo-Christian concept; it’s roughly what everyone must have believed since before people were people. The hard part of grasping evolution was to realize that species weren’t, as they seem to be, unchanging, but had instead evolved from different, simpler organisms over unimaginably long periods of time.</p><p>Now we don’t have to make that leap. No one in an industrialized country encounters the idea of evolution for the first time as an adult. Everyone’s taught about it as a child, either as truth or heresy.</p><p>[2] Philosophy is like math’s ne’er-do-well brother. It was born when Plato and Aristotle looked at the works of their predecessors and said in effect “why can’t you be more like your brother?” Russell was still saying the same thing 2300 years later.</p><p>Math is the precise half of the most abstract ideas, and philosophy the imprecise half. It’s probably inevitable that philosophy will suffer by comparison, because there’s no lower bound to its precision. Bad math is merely boring, whereas bad philosophy is nonsense. And yet there are some good ideas in the imprecise half.</p><p>[3] Aristotle’s best work was in logic and zoology, both of which he can be said to have invented. But the most dramatic departure from his predecessors was a new, much more analytical style of thinking. He was arguably the first scientist.</p><p>[4] The meaning of the word “philosophy” has changed over time. In ancient times it covered a broad range of topics, comparable in scope to our “scholarship” (though without the methodological implications). Even as late as Newton’s time it included what we now call “science.” But core of the subject today is still what seemed to Aristotle the core: the attempt to discover the most general truths.</p><p>Aristotle didn’t call this “metaphysics.” That name got assigned to it because the books we now call the Metaphysics came after (meta = after) the Physics in the standard edition of Aristotle’s works compiled by Andronicus of Rhodes three centuries later. What we call “metaphysics” Aristotle called “first philosophy.”</p><p>[5] Sokal, Alan, “Transgressing the Boundaries: Toward a Transformative Hermeneutics of Quantum Gravity”.<br>[6] Frankfurt, Harry, On Bullshit, Princeton University Press.</p><p>[7] Some introductions to philosophy now take the line that philosophy is worth studying as a process rather than for any particular truths you’ll learn. The philosophers whose works they cover would be rolling in their graves at that. They hoped they were doing more than serving as examples of how to argue: they hoped they were getting results. Most were wrong, but it doesn’t seem an impossible hope.</p><p>This argument seems to me like someone in 1500 looking at the lack of results achieved by alchemy and saying its value was as a process. No, they were going about it wrong. It turns out it is possible to transmute lead into gold (though not economically at current energy prices), but the route to that knowledge was to backtrack and try another approach.</p><p><em>Originally published at </em><a href="http://dsci.blogspot.com/2008/07/shall-we-talk-philospphy.html"><em>http://dsci.blogspot.com</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cffa32fad4f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Tech Debts, Silos and Engineering Culture]]></title>
            <link>https://medium.com/@venkatv/tech-debts-silos-and-engineering-culture-371210fc8f43?source=rss-e9b3bbbc82dd------2</link>
            <guid isPermaLink="false">https://medium.com/p/371210fc8f43</guid>
            <dc:creator><![CDATA[Venkat Vaidhyanathan]]></dc:creator>
            <pubDate>Thu, 13 Feb 2020 06:38:10 GMT</pubDate>
            <atom:updated>2020-02-13T06:38:35.560Z</atom:updated>
            <content:encoded><![CDATA[<p><strong><em>“If the heart doesn’t find a perfect rhyme with the head, then your passion means nothing.” - John Doerr</em></strong></p><p>I am not sure how many of you have seen the traffic evolution in Bengaluru. I have been in this city for the past 15 years and things have changed, quite dramatically. A <a href="https://bangaloremirror.indiatimes.com/bangalore/others/bengaluru-is-the-most-traffic-congested-city-in-the-world-report/articleshow/73698805.cms">recent survey</a> records the city as the “Most traffic congested city”. Clearly, if you have traveled through silk board or privileged like me have to travel everyday through inner ring road, you will notice that this road has been perennially under construction for the last few years. For many of them traveling to say HSR layout from indiranagar, this is the only best route. The road is filled with potholes and massive cracks. Cars and bikes race back and forth, to get to their daily life, work, jobs, families and other things. There always seems to be some or the other construction going on and it doesn’t actually improve. The place is also a low lying area, which constantly gets into flood with a small downpour.</p><p>Once in a while, the municipality(BBMP) throws up barricades, stops traffic one way, fixes things and move onto other things. Worse, they end up cutting more trees, in the name of road expansion, when it is not even needed(to cut trees). White topping roads, among a bunch of unplanned activities, are constantly executed all over the city. The growing traffic and population increase doesn’t stop either. More and more newer companies are coming in, more and more freshers from colleges are recruited, folks from other parts of the country flock in happily to join their new jobs, among other things.</p><p>Unfortunately, a lot of companies actually build “software” this way. They quickly build things, based on the tools and techniques that come their way, ignoring the upcoming disaster and the anticipated traffic that is expected to come along their way. They plant some barricades(rate limits, throttles etc) that make things look nice and move on to another task. Once in a while the road(or software in this case) starts deteriorating and and they end up patching the potholes. The city(or in this case the company) doesn’t have the budget to repave it all at once. Esp, for a startup, the analogy is something like this: “You are driving a car at 100 miles an hour. You need to change the engine, without stopping the car or slowing it down or from causing an accident”. In essence, you patch a few things, plant some flowers or artificial trees along the way and move on.</p><p>There are a multiple ways to look at the above, depending on what vantage point you look at it from. Unfortunately, they all boil down to the same thing. At the <em>individual</em> level, what you often see is <a href="https://en.wikipedia.org/wiki/Law_of_triviality">bikeshedding</a>. That is, engineers focusing time and energy on technical nitty-grities that, in the grand scheme of things, doesn’t matter. Its often focusing on aesthetics or what someone “<em>sees</em>”, than actually focusing on things, that often move the needle, in an impactful and meaningful way. Sometimes, we are so deep caught up with a problem, that we often plant more flowers(or apply more bandage, in this case). When you really drowning in a swamp, filled with crocodiles, it often gets difficult to remember, that you entered the swamp, in the first place, to clean it up. This is often, a function of the manager or leader, who lacks context or doesn’t provide the requisite direction to the team.</p><p>And so, we start breaking the problem down into smaller problems(divide and conquer). What does this mean? We start forming multiple pods, divisions, BUs or other manifestations of a group of people(a team). At a <em>team </em>level, we start tackling these into specific problems(silos). This is an ongoing journey every small to large company takes up. We have different functions, focusing on little parts of <em>their world, </em>almost, in many cases, neglecting the other parts. Engineering focuses on engineering, operations on operations, security on security and so on and so forth. Each of these teams, is fastened so tightly into their respective silos, that it forms a deadlock. There is utter inability to make <em>meaningful </em>progress. The focus of the team is to just get their shit done. Worse yet, what gets done, many a times, is a patchwork of competing, non-negotiable goals and agendas. Its like building new highways on a crumbling road. Even worse, silos within silos(engineering team X vs engineering team Y). Silos are transitive(and perhaps even recursive). Its a natural team evolution, that occurs as organizations grow in accordance with <a href="https://en.wikipedia.org/wiki/Dunbar&#39;s_number">Dunbar’s number</a>. What is needed, is a <strong>cohesive vision</strong>.</p><p>At the <em>organization</em> level, we see large-scale strategy problems and lots of “<em>WIP</em>-lash” — lots of WIP (WIP-lash, pun intended). There is enough of shifting priorities, and lots of “high-priority” items. At times(most times),<em> everything is priority</em>. In certain cases, work and strategy could be devised, 12 months in advance. But, by the time we get down to execute it, the goal post has significantly changed, moved or sometimes even lost. Management is forced to make knee-jerk mandates, in sometimes absolute terms to respond to the newest fire(time for those new flowers to be planted). What does this lead to? <strong>Tech-Debt</strong>!</p><p>Tech debt starts piling up, as things are added to the never-ending priority queue(and tech debts are generally the ones that gets the least priority, among them all). The infrastructure, is in a constant state of mess and ruin, crumbling, like the roman empire. <em>WIP</em>-lash is just strategic bikeshedding.</p><p>What do we do? We, of-course, cannot close the entire road and repave it from scratch. Instead, we can adopt, to do it, in phases. Managing tech debt, works the same way. You pay, when its due. Period! And the payment, becomes due, periodically. We cannot fix it with constant band-aids, double edged tapes and plasters. We have to prioritize it, like everything else. Sometimes, that means, saying <em>NO </em>to other things, we deem important. All of the above, again boils down to your individual company. If your company, isn’t engineering focused, it possibly doesn’t matter. Just like how a state government wants to operate. Some governments focus on economy, while others on agriculture and others on pure infrastructure. In general, a <strong><em>cohesive vision</em></strong> again, at a different level.</p><p>And how do we achieve this cohesive vision? We start with the most macro problems and then go deep down into the micro ones. At the bottom line, many teams and companies have gone to sprints, scrums, agile, etc. The newest kid on the block is the <a href="https://blog.realkinetic.com/okr-process-489891e6b6a8">OKR</a> model. One of the major advantages of starting to work with OKR, is that you cut out the BS and start looking at what’s really happening. You move from an <em>output-oriented</em> towards an <em>outcome-oriented</em> organization. However, OKRs are generally hard to get. But when got right, it provides junior managers, a mechanism to push back and with force, certain hard decisions, that helps open a dialogue between different competing groups. The key discussion on what <em>really </em>matters and what is <em>really</em> possible. To make it more dramatic, lets look at the following dialogue:</p><p><em>A: “Do you want X or Y? I have resources only for one of them.”</em><br>B: <em>“Sure, can we get X to work completely? If done, we can possibly look at achieving part of Y? </em><br><em>A: “Sure, we can get to complete X. But, do not count on completion of Y. If that happens, its completely by chance. “</em></p><p>OKRs are a vehicle for <em>strategic discussions. </em>It is not and i repeat not, tactical status updates, task lists, or waterfall plans. Without some sort of guiding vision that you’re working towards something, you’re just doing stuff. That might <em>look and feel</em> productive, but only on the surface. It <em>must</em> be a negotiation if you want <em>results</em> and not just activity. It’s particularly dangerous in companies that separate product <em>management </em>(“the business”) from product <em>development </em>(“engineering”). At the organization level, we have <em>strategic</em> bikeshedding - being unable to make hard decisions and focus in on what matters to the business <em>right now</em>, resulting in <em>WIP</em>-lash. This is <em>also</em> a prioritization problem, and it leads to the tactical bikeshedding mentioned earlier. In between, at the team level, we have silo-ing. This causes all sorts of issues ranging from deadlock and broken customer experiences to duplication of effort. In effect, the engineering team starts developing identity crisis and alignment issues( <em>Am i even really solving something that is important to the company?)</em></p><p>And now, lets address the elephant in the engineering room — Tech Debt. It all starts with measurement based on impact, severity and other factors. For instance, how do i know when my app is going to break down? Is my app reasonably secure? Does it give the right functional guarantees? Does it handle concurrency well? Well, like i mentioned, we start with metrics. But what are these metrics? That largely depends on your use case. For instance, if you are building a traditional e-commerce app, you will focus on TTB, image loading time, compression, backend cache, front end cache, CDN among a bunch of things. If you are someone like me, working with transactional systems, the problems differ.</p><p>However, like with most things, we are constantly also afraid to accept the reality. It <em>works now</em> is different from it <em>works well. </em>Go back to my road construction analogy at the beginning of this article and you will know why. In essence, avoid <a href="https://techcrunch.com/2011/07/30/vanity-metrics/">vanity metrics</a>.</p><p>And finally, it all boils down to the kind of engineering team you want to build. If you want to build a <a href="https://en.wikipedia.org/wiki/Good_to_Great">great engineering company</a>, behave like one. You cannot become great overnight. It has to be built as part of the company’s DNA. And not all successful companies are great companies. It all starts with <em>your </em>engineering culture!</p><p>On closing note, there isn’t a simple/quick solution to these problems. If leadership and the junior most engineering manager is not in alignment and unable to prioritize what matters, no one else will. Work will continue as usual. And to someone who has joined new, it can look quite reassuring too. But, its <em>culture</em> that matters. OKRs aren’t panacea or silver bullet. They are time consuming and hard to get right. But, if executed well, they can serve as a powerful lens to provide a shared vision. However, from the horse’s mouth on creating the OKR, former intel CEO Andy Grove once said : “The most powerful tool of all is the word <strong><em>NO</em></strong>”.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=371210fc8f43" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>