<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Divij Vaidya on Medium]]></title>
        <description><![CDATA[Stories by Divij Vaidya on Medium]]></description>
        <link>https://medium.com/@divijvaidya?source=rss-92a803c744e4------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*eMa35u8D-UE-dCiVvcWQPQ.jpeg</url>
            <title>Stories by Divij Vaidya on Medium</title>
            <link>https://medium.com/@divijvaidya?source=rss-92a803c744e4------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 10:07:55 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@divijvaidya/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Why monitoring your big data analytics pipeline is important (and how to get there)]]></title>
            <link>https://medium.com/@divijvaidya/why-monitoring-your-big-data-analytics-pipeline-is-important-and-how-to-get-there-a1bc5016f941?source=rss-92a803c744e4------2</link>
            <guid isPermaLink="false">https://medium.com/p/a1bc5016f941</guid>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[monitoring]]></category>
            <category><![CDATA[observability]]></category>
            <dc:creator><![CDATA[Divij Vaidya]]></dc:creator>
            <pubDate>Wed, 29 Jul 2020 00:33:42 GMT</pubDate>
            <atom:updated>2020-07-30T00:39:13.142Z</atom:updated>
            <cc:license>http://creativecommons.org/licenses/by/4.0/</cc:license>
            <content:encoded><![CDATA[<h4><em>Your Big Data analytics pipeline provides insights to your business, now gain insights into your Big Data pipeline itself.</em></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/973/1*0yWiQNnNbbXKONn_HFG2Mg.png" /><figcaption>(image created by author)</figcaption></figure><p>In this article, we first understand the requirements for monitoring your big data analytics pipeline and then we go into the key aspects that you need to consider to build a system that provides holistic observability.</p><p><em>Note: This article assumes some basic familiarity with the big data landscape and associated technologies such as Hadoop, YARN, Spark, Presto etc.</em></p><h3>Background</h3><p>In the technology fueled dynamic environment of the 21st century, for a business to be successful, it needs to learn from the past results, adapt to the current requirements and forecast the future trends. These valuable insights into the past, present and future are retrieved by analyzing vast amounts (large scale) of data collected over time.</p><p>Apart from these, COVID pandemic has taught us that a successful business should be nimble and innovative. Nimbleness comes from the confidence to make the right decision at the right time. Innovation comes from increasing the likelihood of getting lucky by consistently performing experiments. Acquiring both these traits require us to analyze large scale data which could be processed quickly and economically. Today, a growing ecosystem of big data technologies such as Spark, Hive, Kafka, Presto etc. fulfill these requirements.</p><p>The internet is filled with articles about usage of data processing and analytics frameworks. But, there is a shortage of commentary about the auxiliary systems that augment the power of big data technologies. These systems are force multipliers for a better user experience in the big data ecosystem. They provide infrastructure cost attribution, data set discovery, data quality guarantees, audit user access and extract insights (aka gain <em>observability</em>) into your analytical applications which run on the big data stack (henceforth called<em> big data applications</em>).</p><p>This article covers some key ideas to gain observability into big data applications. We would define the scope of requirements, present some known challenges and discuss high level solutions.</p><h3>Introduction to Data Application Observability</h3><p>Big data stack can be broadly divided into the following layers starting from the top of the hierarchy to the bottom: Query processing, execution framework, resource manager &amp; storage. These independent sub-systems work together to execute an analytical query (big data application).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TRhxa-hm6oiusFOoC0ec3A.png" /><figcaption>Layers in the full stack for a big data application — (<a href="https://github.com/apache/incubator-hivemall/blob/master/src/site/resources/images/hivemall_overview_bg.png">Image source</a>)</figcaption></figure><p>Data application observability is the ability to extract insights on the characteristics of big data applications. These insights lead to <strong>lower infrastructure cost</strong> by improving resource utilization; <strong>increase user productivity</strong><em> </em>by ease of debugging and troubleshooting; <strong>improve availability</strong> by reducing mean time to recovery and, <strong>boost security</strong> by providing auditing and accountability.</p><h3>User Personas</h3><p>The user personas for the data application observability can be broadly bucketed into the categories described below. Each kind of persona has different requirement for data freshness, query latency, type and granularity of information.</p><p><strong>End User</strong></p><p>End user is a person who runs queries on the data platform. They could be running an ad hoc query on data using technology such as Presto, Hive, Impala, Spark SQL etc. or could be building a Business Intelligence (BI) dashboard. The job family for this user could be data scientist, software engineer or analyst. This category of user requires metrics &amp; data (henceforth referred as observability statistics) for troubleshooting failures or optimizing performance of their jobs. Some examples of questions that this user persona asks are:</p><figure><img alt="Person looking at a laptop writing hive query and speaking in terms of charges and numbers" src="https://cdn-images-1.medium.com/max/1024/1*6IOC8PCmbsPRKadCaDgW2w.png" /><figcaption>User persona: “End user” writing hive queries to extract insights from data — (<a href="https://favpng.com/png_view/analyst-hd-data-analysis-analyst-big-data-data-science-png/jccnNeZC#_=_">Image Source</a>)</figcaption></figure><p><em>What is the execution status of my query?</em></p><p><em>How can I make my query run faster?</em></p><p><em>What is the most ideal time of the day to run my query based on the traffic in my team’s YARN queue?</em></p><p><em>How much does running this query cost my organization?</em></p><p><em>Is the behavior of this query different from the past? Why?</em></p><p><em>What is the internal execution plan for this query?</em></p><p><em>How can I debug and fix my failed query?</em></p><p><em>Is the allocated capacity to this query sufficient enough to satisfy the query SLA?</em></p><p><em>Where can I find the results of a query that I ran in the past?</em></p><p><strong>Local Resource Administrator</strong></p><p>Local Resource Administrator is the person who is the administrator of infrastructure resources at org/team/project level. In a large organization, it is important to perform cost attribution to various projects/teams so that the return of investment (ROI) could be analyzed. Local resource administrators ensure that the ROI is favorable for their org/team/project.<br>This user persona requires observability statistics for planning infrastructure capacity and maintaining cost efficiency for their organization. They are interested in finding answers to following types of questions:</p><figure><img alt="A person monitoring the cost of infrastructure resources" src="https://cdn-images-1.medium.com/max/640/1*iMSxbRTYIIk0bCvULaU8gg.png" /><figcaption>User persona: “Local Resource Administrator” monitoring the usage of infrastructure across organization — (<a href="https://pixabay.com/vectors/agenda-comic-characters-gantt-2025762/">Image source</a>)</figcaption></figure><p><em>What is the utilization of the allocation YARN queue for my organization?</em></p><p><em>Who (user) is running the most expensive queries on a YARN queue?</em></p><p><em>How much is the total cost to my org for usage of the big data infrastructure?</em></p><p><em>Which app is creating a large number of small files in HDFS?</em></p><p><em>How much capacity should we reserve for this new project?</em></p><p><em>What is the right amount of capacity reservation to meet the SLA?</em></p><p><em>How many applications have failed in the past month because of resource limitations?<br>I am running out of space in HDFS. Do I have any unused data that I can delete?</em></p><p><em>What are the top 10 applications consuming the most CPU/memory in my YARN queue?</em></p><p><em>What are all the applications running in the YARN queue?</em></p><p><em>How many queries did this user run in the last 24 hours?</em></p><p><em>Which are the most accessed tables from this YARN queue?</em></p><p><em>What are the applications waiting for resource allocation in this YARN queue?</em></p><p><em>What are the queries running in this queue sorted by longest time?</em></p><p><strong>Oncall operator</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7cvqaGKjKhIGLfy6WY3ocg.png" /><figcaption>User persona: “Oncall operator” fixing high priority critical incidents - (<a href="https://www.pngfuel.com/free-png/njwfx">Image Source</a>)</figcaption></figure><p>An oncall operator is responsible for responding to critical incidents which might impact the availability or performance or reliability of their service. Their objective is to mitigate the problem as early as possible (reduce MTTR) with minimal impact to the end users.</p><p>To effectively troubleshoot and debug the incident, oncall requires visibility into the current state of the systems at a service level granularity, cluster level granularity and the ability to drill into a query level granularity. They are interested in anomaly metrics to detect the potential cause of inconsistency in the system. Some examples of questions that they ask are:</p><p><em>The system is running out of memory. What is the rogue query that is consuming the majority of memory?</em></p><p><em>This application didn’t make any progress in the last one hour. Where is it stuck?</em></p><p><em>What is the SLA miss rate per Hadoop cluster?</em></p><p><em>How many applications are failing per Hadoop cluster?<br>What are all the queries running in this cluster sorted by longest time?</em></p><p><em>What are all the queries running in this cluster?</em></p><h3>Use Cases</h3><p>Big data applications are complex. A typical workload goes through multiple distributed systems across the depth of the stack. These systems coordinate together for processing a single big data workload. Due to the involvement of numerous complex systems, operational use cases such as troubleshooting failures, identifying performance bottlenecks, finding security holes etc. become hard problems to solve.</p><p>Based on the requirements of different kinds of user persona, the requirement for observability of big data applications can be classified into different use cases. To illustrate the complexity associated with the use cases, we would use an example of big data workload.</p><p>Consider a spark application which is started from a <a href="https://jupyter.org/">Jupyter notebook</a> by a data scientist. The application reads data from two different Hive tables, performs some business logic and then writes the result to a CSV file. The big data stack involved in processing the application is Jupyter Notebook -&gt; <a href="https://spark.apache.org/docs/latest/api/python/index.html">PySpark</a> -&gt; <a href="https://livy.apache.org/">Apache Livy</a> -&gt; Spark -&gt; YARN -&gt; HDFS.</p><p><strong>Application Failure Troubleshooting</strong></p><p>In the example application failure might occur at any layer. Let’s assume that due to incorrect permissions on the HDFS folder containing dependencies for the Spark application, Livy is not able to download (read) the files configured via — py-files of spark-submit. This leads to an application failure. To begin investigating the root cause of the failure, the user would require visibility into the application characteristics such as:</p><ul><li>full software stack for the application so that they can understand that there is a service like Livy which comes into the picture somewhere in the middle of the application execution stack,</li><li>application logs at each layer so that they can determine that error occurred at Livy</li><li>parameter that were passed to Livy so that user can determine what files was Livy trying to access from HDFS</li></ul><p>This example illustrates the difficulty experienced by a user when they troubleshoot a failure in the big data application.</p><blockquote>Users rely on the observability insights to dive deeper into the characteristics of their applications and analyze the causes for failures. These insights help them understand the complete picture of what is happening with the workload from the top level viewpoint of a workload.</blockquote><p><strong>Application Performance Optimization (Cost Efficiency)</strong></p><figure><img alt="A speedometer like device to measure performance" src="https://cdn-images-1.medium.com/max/256/1*2La8LqQHVkqLE6O3u-5dNw.png" /><figcaption><a href="https://www.flaticon.com/authors/vectors-market">Image source</a></figcaption></figure><p>In the example application, let’s assume a scenario when the YARN queue assigned to the spark application is overloaded and as a result, the application is queued for a long time before it starts to execute. From our end user’s perspective the overall application is running slower than usual. To begin troubleshooting the performance bottleneck, user would require visibility into the application characteristics such as:</p><ul><li>full software stack for the application so that they can understand that the execution is stuck at YARN layer,</li><li>execution breakdown at each layer of the stack so that they can understand that upstream layers from YARN processed the application in usual time,</li><li>local execution status at each layer so that they can understand that the application is in queued state at YARN</li><li>resource utilization at each layer so that they can understand that the application is asking for 1000 executors from YARN which are not currently available in the queue</li></ul><p>Additionally the user would need metadata such as which YARN queue was assigned to the application, what were the configuration parameters passed to Spark etc. Although this hypothetical scenario exhibits a simple performance bottleneck, it illustrates the kind of granular observability required to troubleshoot performance in a big data application.</p><blockquote>Users rely on the observability insights to dive deeper into the characteristics of their applications to find sub optimal configurations or performance bottlenecks. Optimizing these bottlenecks result in faster applications which in turn improves the cost efficiency of the underlying hardware resources.</blockquote><p><strong>Application lineage</strong></p><p>In the example application, for any troubleshooting or monitoring, the user has to correlate the notebook identifier with the Livy session ID which should be mapped to the spark application ID which in turn should be mapped to YARN application ID.</p><blockquote>Users of other auxiliary big data systems such as such as user access auditing or data lineage rely on the observability insights to determine the application lineage across the depth of the big data stack.</blockquote><p>As an example, to find the users which have accessed a particular Hive table, mapping is required between the HDFS NameNode API calls to the Spark Application ID.</p><p><strong>Resource management</strong></p><p>Fair allocation of resources amongst big data application of different priority is achieved through abstractions such as YARN queues or Hadoop clusters.</p><blockquote>Observability into the utilization of YARN queues and Hadoop clusters is essential to ensure a fair distribution of resources across users.</blockquote><p>It can prevent scenarios such as critical jobs suffering from resource starvation, resource hogging by a single user, runaway query consuming majority of resources etc.</p><h3>Challenges of building a BigData Application Observability system</h3><p><strong>Collection and publishing of metrics is not a standard feature in big data systems</strong>. The degree of support for collecting and emitting metrics varies across a spectrum. There are systems which collect the metrics and expose an API to expose them such as YARN, systems which provide hooks into the execution lifecycle of a query such as Hive &amp; Spark and systems like HDFS that do not provide any significant monitoring feature out of the box. Depending on the set of systems used in your organization, getting end to end holistic observability into an application might require devising custom solutions for collecting and publishing the metrics. Writing these custom solutions requires an in-depth understanding of the otherwise opaque (it-just-works) open source system since an inefficient metric collection code can significantly hamper the execution performance of the system.</p><p>Processing the metrics collected from these heterogeneous systems becomes complex due to <strong>lack of a common vocabulary</strong> to define the metrics. Each system across the depth of the stack emits the metrics as loosely typed strings or integers. Extensive raw data processing and normalization is required to provide holistic cross stack visibility to the users.</p><p>The <strong>scale of the metrics growth is not linear </strong>to the growth in the number of jobs. Each job emits various kinds of metrics which can fan out the scale to an exponential growth trajectory. The observability system should be able to handle the scale with realtime latencies.</p><p>Depending on the open source platform, the<strong> metrics retention on the server varies from a few minutes to a few hours</strong>. If the metrics are not collected within this duration, they are lost forever. The observability system should be highly available and robust in its collection mechanism as an availability drop of a few minutes could lead to permanent data loss.</p><p><strong>Traditional observability systems designed for web services do not satisfy the requirements of providing observability for big data applications</strong>. Cardinality of metrics, kinds of metrics and the type of signals other than metrics for a big data application varies as execution moves from one sub-system to another. Traditional observability systems support limited cardinality (distinct set of values) and limited max number of dimensions for a metric. In case of big data, as an example Spark alone has <a href="https://spark.apache.org/docs/latest/configuration.html">more than 350 configuration parameters</a> which could be used as dimensions. An ideal big data application observability system would be able to correlate the metrics emitted across multiple stacks to provide a unified view of the execution model for the application.</p><p>Lastly, tracing the application across the stack layers is a challenge due to the <strong>lack of universal tracing</strong> amongst open source applications. Each layer in the stack creates its own unique ID and it is left to an external system to map the IDs with each other. As an example, a hive query might have an ID such as hive_123 while the YARN application ID would be application_234 and Livy batch ID might be batch_789. But these all IDs refer to the same hive query which uses Apache Livy &amp; Spark behind the scenes for execution. There are some recent efforts to integrate <a href="https://www.jaegertracing.io/">Jaeger</a> with Hadoop in the open source world but the universal tracing hasn’t covered the entire depth of the stack yet.</p><h3>Solving Data Application Observability</h3><p>There are three milestones in the journey towards providing observability for data applications. Each milestone is a necessary building block for the next and provides incremental value to the user.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1007/1*NY9bDmp8Y_iStaD7QwumKw.png" /><figcaption>Milestones towards holistic Data Application Observability — (Image created by author)</figcaption></figure><h4>Milestone 1: Visibility into application characteristics</h4><p>At the completion of this milestone, users would get granular visibility into the raw metrics associated with the lifecycle of an application. The task of interpreting the metrics and extracting useful insights is left up to the user.</p><p>From a technical standpoint, this milestone creates a number of building block components of the entire observability system. This article would explore three high level components without going into technical implementation details (currently, there is no complete open source implementation which could be cited as an example here).</p><p><strong>The user facing component should allow users to consume information both visually (UI) and programmatically(APIs)</strong>. An intuitive user interface ties together different metrics collected from across the stack to provide holistic observability into the application. A user of the interface might not necessarily know under the hood details of the application and thus, the user experience should cater to users of different technical acumen. As an example, a marketing executive running a Hive SQL query to get results for a campaign does know that the Hive query was executed on a YARN cluster using the Spark processing engine. They are only interested in checking whether their query has completed from the interface. On the other end of the spectrum, the engineer from the Hive team would be interested in exploring the hive query execution plan or looking at the metrics of YARN containers when the query was being executed. Some open source systems ship with their own in-built UI such as <a href="https://spark.apache.org/docs/latest/web-ui.html">Spark Web UI</a> or <a href="https://hadoop.apache.org/docs/r3.0.1/hadoop-project-dist/hadoop-common/ClusterSetup.html#Web_Interfaces">Hadoop Web UI</a> or <a href="https://tez.apache.org/tez-ui.html">Tez UI</a>. These interfaces are cater to the users who have in-depth knowledge of these systems. The author of this article does not recommend taking user experience inspiration from these UIs for observability systems which cater to the user personas discussed earlier in the article.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*v0l38Yv3mMKokTI5" /><figcaption>Open source Spark web UI — <a href="https://raw.githubusercontent.com/japila-books/apache-spark-internals/master/modules/webui/assets/images/spark-webui-jobs.png">Image source</a></figcaption></figure><p><strong>Collectors perform realtime collections of metrics across different layers of stack</strong>. They guarantee ordering of lifecycle events and at-least-once collection of events. We have discussed the challenges associated with collectors earlier in the article. Custom implementation is required to leverage hive hook or spark hook which could provide basic lifecycle metadata for an application, you could write custom periodic collectors to collect information from <a href="https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html#Cluster_Metrics_API">YARN APIs</a>, devise custom agents to parse and publish the <a href="https://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/apidocs/org/apache/hadoop/yarn/server/nodemanager/NMAuditLogger.html">HDFS audit logs</a> and writing custom agents to collect the extract CPU/Memory utilization for spark executors. Although multiple software as a service (SaaS) providers provide functionality described here, in the open source domain, very limited support is available for these collectors which could be leveraged as example implementations such as <a href="http://atlas.apache.org/#/HookHive">Apache Atlas</a> for hive hooks, <a href="https://github.com/apache/ambari/blob/trunk/ambari-server/docs/api/v1/index.md#monitoring">Apache Ambari</a> for basic cluster monitoring or <a href="https://eng.uber.com/jvm-profiler/">Uber JVM profiler</a> for resource consumptions by spark executors.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KHzkETytJjv12YEeLif2Tg.png" /><figcaption>Milestone 1: Components of Data Application Observability system — (Image created by author)</figcaption></figure><p><strong>Pipelines for metrics enrichment bridge the gap between the user interface and the metric collectors</strong>. This component ingests the metrics published by the collectors, processes the metrics and stores them so that the user interfaces could read them. This component should be able to handle the scale of metrics being ingested into the system and should have the ability to process each metric without impacting the metric propagation time to the end user. The requirement for metric propagation time to end user aka metric freshness varies from use case to use case. For some use cases such as alerting when an application is consuming 90% of YARN queue resources, the metric freshness should be in the order of seconds while use cases such as performance optimization could be solved by a metric freshness in the order of minutes. The processing phase of the metrics involves cleaning the metrics, normalizing the metrics into a standard vocabulary and enriching the data by gathering auxiliary information from alternate sources. Some examples of processing &amp; enrichment tasks are, uploading the query plan size of 1MB to a blob store instead of passing it on the wire across the pipelines; enriching the HDFS audit logs by retrieving the mapping between hive table and HDFS files from the Hive MetaStore (HMS); joining the information from different metrics streams to create an application lineage; calculate cost of a job based on utilization metrics etc.</p><h4>Milestone 2: Actionable insights</h4><p>At this stage, the system would provide users with insights derived from the raw data. This would save the user effort to sift through the plethora of raw metrics to find the significant metrics. The system is able to suggest actions for the user because it understands the underlying correlation of the metrics.</p><p>In the previous milestone, the intuitive user interface displays complete end to end observability to the user. At this milestone this interface changes from a research driven interface to a workflow based interface. A research driven interface would display all the possible metrics to the user and let the user interpret them while the workflow based interface would enter the user into a funnel related to an anomaly, at the end which the anomaly would have been analyzed and if required, fixed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vEPrta4irNqjBrM4Zfz_XQ.png" /><figcaption>Milestone 2: Components of Data Application Observability system — (Image created by author)</figcaption></figure><p><strong>A new “insight extraction” component is added to the metric processing pipelines which is triggered on receiving critical events such as lifecycle state change</strong>. This pipeline runs a set of rules to determine if a metric is anomalous or not. These rules could either be static or artificially generated using a Machine Learning model. <br>The full implementation for such a complex component is out of the scope for this article but to demonstrate a proof of concept, let us revisit the example we discussed earlier in the article (Use Cases section) where an application might run slower than usual or time out due to YARN queue overload. On receiving periodic events associated with queued time for an application, this new component would compare it against the average queue time for an application in the YARN queue which is derived based on historical information. If the current queue time is higher, this system could <a href="https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html#Cluster_Scheduler_API">retrieve the current YARN queue</a> utilization to derive the top 5 longest running applications on the queue and top 5 applications consuming the most resources. Based on this data, the user could be entered into a workflow in the interface where they would be given options to either kill one of the top 5 apps in the queue or schedule their job to run at a different time suggested by the component based on historical queue usage pattern.</p><h4>Milestone 3: Self healing</h4><p>At this stage, the system would perform the actions that it has derived in Milestone#2 instead of relying on the end user to manually perform the actions. The results of the analysis performed by the “insight extraction” component acts as a feedback loop to the big data frameworks. Based on the feedback, these frameworks automatically tune the parameters for application.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DRaKaoG7mhJAIgc-uyO8YQ.png" /><figcaption>Milestone 3: Components of Data Application Observability system — (Image created by author)</figcaption></figure><p>To illustrate the functionality let us consider an example where you run a spark job and it runs slowly. Achievements in milestone#1 provides you with the information about the execution timeline of the spark job. The systems developed in milestone#2 will highlight the longer than usual execution time. Specifically, it suggests that the majority of execution time is spent on writing results to HDFS. To remedy this, system would also suggest that you set the repartition flag to 1. During this milestone, the system would provide feedback to the spark execution engine so that future runs of this application will be executed with the repartition flag set to 1. This leads to self healing for the application without any user intervention.</p><h3>Conclusion</h3><p>With the data processing systems becoming critical to modern online businesses, understanding the characteristics of big data applications is crucial to maintain a robust, efficient, secure, easy to use and performant data analytics platform. Hopefully this article has provided you with a glimpse into the opportunities that observability into your data applications could unlock and guided you towards a path in the journey to achieve the desired objective.</p><p>Disclaimer: All opinions and thoughts expressed in the article are my own and do not reflect the opinions of my past or present employer.</p><p><em>If you like reading about tech musings, follow me on Twitter </em><a href="https://twitter.com/divijvaidya"><em>@divijvaidya</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a1bc5016f941" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Amazon to Uber: From the lens of a software engineer]]></title>
            <link>https://medium.com/swlh/amazon-to-uber-from-the-lens-of-a-software-engineer-e5bd1c38caba?source=rss-92a803c744e4------2</link>
            <guid isPermaLink="false">https://medium.com/p/e5bd1c38caba</guid>
            <category><![CDATA[amazon]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[uber]]></category>
            <category><![CDATA[software-development]]></category>
            <dc:creator><![CDATA[Divij Vaidya]]></dc:creator>
            <pubDate>Mon, 08 Jul 2019 01:46:21 GMT</pubDate>
            <atom:updated>2020-09-26T23:17:34.603Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sMN_9TDiWV9RjD_nwTFY6w.jpeg" /></figure><h3>Background &amp; Introduction</h3><p>I am a software engineer, who recently switched³ companies from Amazon to Uber. I have worked for 6.5 years at Amazon across multiple organizations and geographies. For the past one month, I have been working with the Data R&amp;D organization of Uber at Seattle.</p><p>This post addresses the question my friends and colleagues have asked me, “What is the difference in working at Amazon vs. Uber?”</p><h3>What this post is not!</h3><p>Engineering culture of a company cannot be defined in broad strokes as it is unique to every team and every org. The thoughts presented in this post are based on my <strong>observation working with specific teams and organizations</strong> and <strong>do not represent the broader company culture</strong>.</p><p>This post <strong>does not reflect upon my rationale to make the first company switch</strong> in my career and hence, the content should not be interpreted as such.</p><h3>From Amazon to Uber</h3><p>There are multiple commonalities and contrasts between the work culture at both the companies. Some of these can be attributed to the relative size of the companies while others are inherent &amp; unique to the DNA of companies themselves. I am going to outline some noteworthy differences that I have encountered.</p><h4><strong>DevOps hustle</strong></h4><p>DevOps hustle is everywhere. Trying to maintain high availability for your services while not impeding rapid pace of innovation and continuous production releases is a hard problem to solve. Even more so, when we are talking about the massive scale at which Amazon and Uber operates. Both companies have a laser sharp focus on operational excellence, but while Amazon has long established, mature processes towards addressing the problem, Uber is still still experimenting with what strategy works best for them.</p><p>Amazon might be notorious (unfairly if I may add, see SIDENOTE#1) for heavy ops burden on software engineers with round the clock pager duty &amp; on-call support, the reality is (IMO) that every company with a little ounce of customer obsession would take the reliability of its systems very seriously and Uber is no exception to that. Hence, DevOps hustle at Uber is no different than at Amazon, maybe even more, owing to rapid growth of the newly minted platforms &amp; loosely structured operational processes.</p><h4><strong>Open source culture</strong></h4><p>Usage of OSS is perhaps the biggest difference that jumped out as a surprise to me when I joined Uber. Uber is an open source first company. This is reflected from its choice of internal tools and technologies to its impressive track record of giving back to the community [see SIDENOTE#2].</p><p>Building the internal infrastructure around open source technologies gives developers freedom to choose the right tool for the job and spend their energy in solving business problems where innovation is required. This also means that you have a diverse network of developers over the internet to help when stuck trying to debug an issue with a third party library. In contrast, working at Amazon which relies mostly on in-house developed infra &amp; tools, narrows down your list of options to choose the correct tools and limits the amount of support options you have when stuck with an issue.</p><p>Instead of building every developer productivity tool itself (for monitoring, dashboarding, paging, recruiting), Uber licenses a lot of tools from companies which provide Software As A Service. These smaller companies with a focused portfolio tend to have better software for their domain that any in-house developed tooling. As a developer, this gives you the latest &amp; the greatest tools to boost your productivity and improve software development lifecycle. This also frees up developers to focus on developing software for business problems &amp; features unique to Uber.</p><h4>Transparency</h4><p>At Uber, there is a weekly hands-on with the company leadership where the execs answer questions from employees. In addition to that there are periodic all-hands at every level of the executive chain.</p><p>As a developer, this gives you transparency around the rationale behind the decisions made that impact you &amp; your work. This helps you align your work more closely with the company objectives and ensures that you are spending your energy trying to solve the right problems. It also gives you visibility into the challenges that exist beyond your immediate team or org. Coupled with the strong emphasis on documentation, this gives you an opportunity to get the answer to every “why” that you might have around any decision (technical or not) made. As a hypothetical example, if I have a question on “Why is Uber leaning towards Go over Java? or Why is my org structured the way it is?”, there would be a document which is accessible to me explaining the reason in great detail.</p><p>Such degree of transparency pertaining to every decision whether technical or not is refreshing and is extremely helpful during on-boarding to your team &amp; understanding the state of the existing infrastructure. In contrast, at Amazon, the level of transparency was exposed on a need to know basis.</p><h4>No AWS</h4><p>I miss the days of getting an auto scaling, fully managed MySQL compatible database on the click on a button. While working at AWS, I underestimated the simplicity that the cloud Infrastructure As A Service (IaaS) has brought in the development lifecycle of a software engineer. Using the cloud services comes at a cost and, at Uber, while you can use any technology that you need to get the job done, you have to justify the cost of paying a cloud vendor vs. managing/developing on your own.</p><h4>Social interaction at workplace</h4><p>Due to the small headcount at the Seattle engineering office of Uber, the work place feels tightly knit where you personally know people other than co-workers in your immediate team. The social events during lunch hours and the cultural site-wide events play an important role in bringing the feeling of togetherness as a unit. In contrast, Amazon operates in small functional units where your social interaction is limited to your two-pizza team (unless you make an extra effort to join company wide events which are always overcrowded).</p><h4>Engineering challenges</h4><p>Both the companies are not very different in the kind of engineering challenges that you would solve. In either case, you would have a business use case which cannot be solved by existing solutions due to the scale and availability requirements (perhaps the requirements for AWS might be greater than Uber’s in different domains). The business problems at these companies might be different but from an engineering perspective, at the end of the day, you are designing systems which work at a larger scale, are more robust, simpler to use and faster than the existing solutions. These solutions are tailored to fit the exact business use case which can be complex and diverse across these companies. Perhaps, the contrast lies in the number of engineers dedicated to solve a problem. A two pizza team working on a problem at Amazon would probably have a single developer working on the solution at Uber.</p><h4>Involvement in strategic decisions &amp; roadmap</h4><p>At Amazon, this is a topic which is heavily influenced by the org you are working on and the direction set by the executive leadership. I have experienced involvement ranging from no visibility into the data behind a top down mandated project requirement to having complete involvement via an engineering led round table discussion for the 2–5 year plans for a technical product. At Uber, every decision associated with the internal product that I own rests with me, right from identifying the gaps to defining success metrics. With my limited exposure at Uber, this seems to be the norm in my org rather than an anomaly (it is a platform org, not a product org) and ties to the point of a single person having more ownership at Uber than at Amazon.</p><h4><strong>Working in a remote office</strong></h4><p>At Amazon, I used to work at the HQ in Seattle where all the stakeholders, partners and upstream dependencies were accessible in person with a 10 minute walk, whereas at Uber, my stakeholders &amp; dependencies are distributed across geographies. This particular point is very specific to the teams I worked with at either companies, but is nevertheless a big change I felt in terms of my work. With remote dependencies and stakeholders, one has to make extra effort to bring together the synergies. Video calls do not fully substitute the advantages of in-person meetings &amp; discussions. Neither do occasional cross office visits replace the home advantage of having everyone you work with within the same office space.</p><h4>Ground up innovation</h4><p>Uber encourages the software engineers to seek and solve problems outside their immediate work domain and team’s charter. If you identify an engineering problem that you want to tackle and it would benefit Uber, the management is very likely to allow you to spend time in solving that problem even if that problem scope is outside the team’s ownership. In contrast, at Amazon, it is difficult to get visibility of the challenges being faced at other orgs and hence, difficult to work on a problem without actually moving to a different team.</p><h4>Lunch/snacks at office (It matters!)</h4><p>Amazon does not provide free lunch to its employees and its offices do not have cafes where you can purchase wholesome lunch daily, while Uber has company provided free catered lunch with rotating daily cuisines. No matter how I try to justify that this difference doesn’t matter in the grand scheme of things, the truth is that, having the option to eat lunch at office &amp; availability of on-demand snacks throughout the day has boosted my productivity. No more spending half hour hunting for food in crowded downtown restaurants or the pangs to go home early because you are feeling hungry.</p><h3>Looking forward</h3><p>As I graduate from being a nUber (this is what new hires are called at Uber) and getting some experience under my belt at Uber, I am starting to appreciate how do the company’s core values/tenets manifest in the day to day work, something which Amazon excelled at. As a developer, these tenets help in the process of decision making and bringing alignment in discordant views across an organization. I would write more about this once I have a better understanding of this and other realization that come only with experience.</p><p>Meanwhile, you can follow me on Twitter to get updates about my software engineering journey at <a href="https://twitter.com/divijvaidya">https://twitter.com/divijvaidya</a></p><p><strong>SIDENOTE#1</strong>: Being oncall for the systems that I own has perhaps been the best teacher for me in my career. This has guided me towards writing operationally sound and secure systems from ground up and I truly believe that every company should adopt Amazon’s model of oncall &amp; operational processes.</p><p><strong>SIDENOTE#2</strong>: Cadence, M3, Ludwig, Fx, Marmaray etc..</p><p><strong>SIDENOTE#3</strong>: As of Aug 2020, I have moved back to Amazon Web Services after a short but eventful journey at Uber.</p><p>Disclaimer: All opinions and thoughts expressed in the article are my own and do not reflect the opinions of my past or present employer</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e5bd1c38caba" width="1" height="1" alt=""><hr><p><a href="https://medium.com/swlh/amazon-to-uber-from-the-lens-of-a-software-engineer-e5bd1c38caba">Amazon to Uber: From the lens of a software engineer</a> was originally published in <a href="https://medium.com/swlh">The Startup</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Lessons from “Snapchat stories on DynamoDB” at re:Invent 2017]]></title>
            <link>https://medium.com/@divijvaidya/lessons-from-snapchat-stories-on-dynamodb-at-re-invent-2017-ccb9e81663a9?source=rss-92a803c744e4------2</link>
            <guid isPermaLink="false">https://medium.com/p/ccb9e81663a9</guid>
            <category><![CDATA[snapchat]]></category>
            <category><![CDATA[software-architecture]]></category>
            <category><![CDATA[dynamodb]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[awsreinvent]]></category>
            <dc:creator><![CDATA[Divij Vaidya]]></dc:creator>
            <pubDate>Fri, 08 Dec 2017 23:01:04 GMT</pubDate>
            <atom:updated>2017-12-08T23:01:04.144Z</atom:updated>
            <content:encoded><![CDATA[<p>Dave Killian from Snapchat presented a case study on migration of one of their backend components to DynamoDB. The talk was presented at re:Invent 2017 and is available on YouTube here: <a href="https://www.youtube.com/watch?v=WUleQzu9l_8">https://www.youtube.com/watch?v=WUleQzu9l_8</a></p><p>Dave binds together an interesting narrative and articulates the lessons learned from the project at the end of the talk. Here, I would expand on few points which I find worth calling out from this case study.</p><ul><li><strong>Validate theoretical assumptions with experiments &amp; proof of concepts</strong>. Rapid innovation thrives in environments where quick decisions are made based on reasonable assumptions and risks. Data from small scale prototypes/experiments help reinforce the confidence behind the decision. In my experience, prototype/experiment is much more useful for convincing people about the merit of a design decision than writing a theoretical design document. This point is not emphasized enough through the formative years of a software engineer.</li><li><strong>Don’t disregard utility of persistent connections</strong>. SSL handshake is expensive, especially when your application is sensitive to a granularity of milli second. In this case study, maximizing the use of persistent connections led to better performance as well as better availability. I am also tempted to talk about WebSockets in this context, but that is a longer conversation for some other time.</li><li><strong>Removing a bottleneck from a system doesn’t eliminate the bottleneck but shifts it to another component.</strong> In this particular case study, the bottleneck shifted from the storage to the task queue. When you have solved a scaling/performance problem, it is important to understand what is the new bottleneck and it’s limitations. Black box stress testing the system from a customer’s perspective would potentially have revealed the new bottleneck in this case.</li><li><strong>Think through your data model for your use case. </strong>The structure of your data plays a key role in defining the processes you would build to reason over the data. You should identify potential data access patterns (corresponding to customer use cases) very early in the architecture design process and model your data storage accordingly. In this case study, as an after thought, the data was normalized to optimize for the most expensive access pattern (mutation on unique story identifier). Moving to the normalized model also reduced the number of contentious points in the system and simplified transactions.</li></ul><p>Have you watched the talk yet? Feel free to post your thoughts in the comments below.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ccb9e81663a9" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Simply explained: Object Oriented software]]></title>
            <link>https://medium.com/@divijvaidya/simply-explained-object-oriented-software-8496cf62de0b?source=rss-92a803c744e4------2</link>
            <guid isPermaLink="false">https://medium.com/p/8496cf62de0b</guid>
            <dc:creator><![CDATA[Divij Vaidya]]></dc:creator>
            <pubDate>Sat, 01 Nov 2014 20:42:20 GMT</pubDate>
            <atom:updated>2014-11-01T20:42:20.160Z</atom:updated>
            <content:encoded><![CDATA[<p>Excerpt from a 1994 <a href="http://www.rollingstone.com/culture/news/steve-jobs-in-1994-the-rolling-stone-interview-20110117">Rolling Stone interview</a>, Jobs explains what object-oriented programming is.</p><blockquote><strong>Jeff Goodell: </strong>Would you explain, in simple terms, exactly what object-oriented software is?</blockquote><blockquote><strong>Steve Jobs: </strong>Objects are like people. They’re living, breathing things that have knowledge inside them about how to do things and have memory inside them so they can remember things. And rather than interacting with them at a very low level, you interact with them at a very high level of abstraction, like we’re doing right here.</blockquote><blockquote>Here’s an example: If I’m your laundry object, you can give me your dirty clothes and send me a message that says, “Can you get my clothes laundered, please.” I happen to know where the best laundry place in San Francisco is. And I speak English, and I have dollars in my pockets. So I go out and hail a taxicab and tell the driver to take me to this place in San Francisco. I go get your clothes laundered, I jump back in the cab, I get back here. I give you your clean clothes and say, “Here are your clean clothes.”</blockquote><blockquote>You have no idea how I did that. You have no knowledge of the laundry place. Maybe you speak French, and you can’t even hail a taxi. You can’t pay for one, you don’t have dollars in your pocket. Yet I knew how to do all of that. And you didn’t have to know any of it. All that complexity was hidden inside of me, and we were able to interact at a very high level of abstraction. That’s what objects are. They encapsulate complexity, and the interfaces to that complexity are high level.</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8496cf62de0b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Exposing your personal data to technology: a win-win scenario]]></title>
            <link>https://medium.com/@divijvaidya/exposing-your-personal-data-to-technology-a-win-win-scenario-1c913652a7b4?source=rss-92a803c744e4------2</link>
            <guid isPermaLink="false">https://medium.com/p/1c913652a7b4</guid>
            <dc:creator><![CDATA[Divij Vaidya]]></dc:creator>
            <pubDate>Tue, 29 Jul 2014 16:36:28 GMT</pubDate>
            <atom:updated>2014-07-29T16:36:28.850Z</atom:updated>
            <content:encoded><![CDATA[<p>There have been many debates about privacy of user data and a lot said over the time, particularly since <a href="http://online.wsj.com/articles/furor-erupts-over-facebook-experiment-on-users-1404085840">FB social experiment</a> was revealed. But here follows what I think:</p><p>Q: What is the future of technology?<br>A: For that we first have to answer what is the purpose of technology. The purpose of technology is to make our lives better.</p><p>Q: That is vague, it is already making our lives better.<br>A: Really? Are your gadgets able to detect your mood and change personalization effects accordingly? Can you fridge refill itself as per your needs? Can you broadcast your thoughts just by thinking? Every individual is different and just like the best of friends know you inside out, for technology to become a friend, it needs to understand you.</p><p>Q: But I don’t want to share my personal life data with a machine!<br>A: Then, technology can never be your best friend. It can never be able to “know you”, hence, cannot give you a unique experience that influences you life in ways unique to you. Without this data, technology would be limited to serving you like an average user with whom you might not share many traits.</p><p>Q: But wouldn’t that data make the person operating the machine super powerful? A case in point is what happened with <a href="http://en.wikipedia.org/wiki/PRISM_(surveillance_program)">prism</a>.<br>A: Now, you are coming to the point where you have a problem with security of your data and not really with sharing your data if it is made secure. You have a valid point. The company should ensure security of data and we should trust them that they are doing this. If we don’t trust them, then we shouldn’t be using their app.</p><p>Q: This app is collecting my data that they don’t need for their features.<br>A: Really? Do you know the feature set they take into consideration when training a machine learning model? Do you know about all the projects using this data to launch a feature in 2015?</p><p>Q: They should at least inform me that they are using my data to train their ML model.<br>A: Companies experiment with features all the time. It is not scalable to put every little experiment in front of the user. One, it would lead to loss of IP, two, it would result in a lengthy T&amp;C type thing that nobody would bother to read. Well, who knows, your app might actually be telling you how they are using your data, did you care to read the T&amp;C?</p><p>Having said that, If I were to build an app it would do the following with respect to collecting user data:<br>1. Ask you for your permission and inform you about what kind of user data it is collecting. Also, inform the user how that data would be used.</p><p>2. May give the option for you to opt out of that data collection. I am saying “may” because if I feel that without that data the user will be served with degraded experience of my app, then, I will choose having no experience at all rather than having bad customer experience. So, for such cases, it must be either you give the data or you can’t use the app.</p><p>3. Ensure that the user confidential information(UCI) is secure. Security of UCI and other user data (might not be confidential) should be paramount and of highest priority in the org.</p><p>4. Using that data for personalizing ads is acceptable as long as we have mentioned that as per point 1. The cost for you to use that *free* app is sharing your user data to display targeted ads for you. Ads, in the end are a tool to help you make informed decision while purchasing things for yourself. I don’t say ads are bad but that’s a debate for another time.</p><p>5. Provide a wipeout option, where all user info would be erased from all systems.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1c913652a7b4" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>