<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Clarisights - Medium]]></title>
        <description><![CDATA[Clarisights is the decision platform for growth. Realtime, interactive and contextual reporting for high performance marketing teams. - Medium]]></description>
        <link>https://medium.com/clarisights?source=rss----284fe0b03aa7---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>Clarisights - Medium</title>
            <link>https://medium.com/clarisights?source=rss----284fe0b03aa7---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 19:33:20 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/clarisights" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Evaluating complex arithmetic expressions at scale]]></title>
            <link>https://medium.com/clarisights/resolving-metric-dependency-expression-with-dag-ast-da4a90fcb7ff?source=rss----284fe0b03aa7---4</link>
            <guid isPermaLink="false">https://medium.com/p/da4a90fcb7ff</guid>
            <category><![CDATA[algorithms]]></category>
            <category><![CDATA[directed-acyclic-graph]]></category>
            <category><![CDATA[abstract-syntax-tree]]></category>
            <category><![CDATA[engineering]]></category>
            <category><![CDATA[data-structures]]></category>
            <dc:creator><![CDATA[Prashant Vithani]]></dc:creator>
            <pubDate>Mon, 01 Apr 2019 06:43:23 GMT</pubDate>
            <atom:updated>2019-04-10T16:58:25.284Z</atom:updated>
            <content:encoded><![CDATA[<p>At Clarisights we are building an automated BI platform. Our goal is to help non-SQL literate business users to ask questions about their data at the deepest granularity without worrying about pipelines, data de-normalization or writing complex queries. We enable customers to create “<a href="https://www.clarisights.com/blog/journey-of-custom-metrics/">custom metrics</a>” which is a way for them to define <a href="https://en.wikipedia.org/wiki/Performance_indicator">KPIs</a> (Key Performance Indicators) on top of base metrics from marketing data sources. These “custom metrics” are used to perform complex arithmetic and conditional operations on metrics across multiple data sources, and users can define them quite easily from the interface.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eAyDefdCNo1GNv0iL88fJQ.jpeg" /><figcaption>A typical custom metric evaluation</figcaption></figure><p>A typical query can have several custom metrics, each depending on other custom metrics and it can touch 10s of millions of rows per data source. The aggregated results from data sources go through 100s of thousands of complex expression evaluation while fulfilling interactive SLAs.</p><p>While it looks deceptively simple in UI, designing its architecture was a really complex problem to solve. We had to go through multiple iterations to get the computations right for nested, cross-channel queries and use multiple data structures such as <a href="https://medium.com/basecs/leveling-up-ones-parsing-game-with-asts-d7a6fc2400ff"><strong>AST</strong> — <strong>A</strong>bstract <strong>S</strong>yntax <strong>T</strong>ree</a> to represent expressions and <a href="https://en.wikipedia.org/wiki/Directed_acyclic_graph"><strong>Directed Acyclic Graph</strong></a> to track dependencies of the metrics on one another.</p><h4>V1 of Custom metrics</h4><p>Our 1st version was extremely basic, It only allowed two operations ‘sum’ and ‘multiplication’ with no parentheses.</p><pre>(a + b) * c had to be written as a * c + b * c</pre><p>To define a ratio, the user had to specify numerator and denominator expressions separately, which is not particularly intuitive e.g. The expression (a / b) * c would be defined as:</p><pre>{ numerator: &quot;a * c&quot;, denominator: &quot;b&quot; }</pre><h4>Performance Problems:</h4><p>To calculate the values of metrics, they had to be broken into their dependencies for computations across multiple data sources. After computing the sum of each individual dependent metric across channels, the ratio is calculated to get the final value. This design failed when there was a need for creating nested custom metrics. If a metric has another custom metric as its dependency, it will report wrong values or zero since we don’t store the custom metrics, but calculate them on the fly i.e. the database has no info about the dependent custom metric.</p><h4>Enter Recursion (V2):</h4><p>We tackled some of the performance problems using recursion. In cases where a custom metric is a dependency, it is in-turn broken down to an expression consisting of the most basic metrics which are stored and aggregated directly in the DB.</p><pre>Consider a calculated metric: cm = cm1 + cm2<br>where cm1 = base_metric1 + base_metric2<br>and   cm2 = base_metric2</pre><pre>base_metric1 and base_metric2 are stored in DB collection</pre><pre>The final expression of cm before the query goes to DB will be</pre><pre>cm = base_metric1 + base_metric2 + base_metric2</pre><p>We used the ruby gem <a href="https://github.com/rubysolo/dentaku">dentaku</a> which is the arithmetic &amp; logical expression evaluation library. The recursion is performed on AST until it reaches the leaf nodes, which are base metrics. The limitations of V1 had also been addressed in this version such as adding support for brackets, deprecation of “numerator” and “denominator” — single string expression, support of all operators (+, -, *, /). In addition to this, on the fly validations were added to call out bad input.</p><p>While this was a significant improvement, as usage increased we found newer use cases which we had not accounted for in metrics, where a custom metric is a combination of base metrics from multiple different channels and is also a ratio. When the metric definition contains a ratio, the computation of the final value of the metric must not be part of the aggregation query as it will yield calculated ratio of a metric which cannot be merged with the ratio of the same metric from another data source. Consider below example:</p><pre>Expression: ROI = revenue / spend</pre><pre>revenue = web revenue + app revenue<br>spend = search spend + social spend</pre><pre><strong>What happens in the V2:<br></strong>- Facebook returns the computed number of revenue/spend; <br>  effectively <em>(web revenue + app revenue)/(social spend)</em></pre><pre>-<strong> </strong>Google Ads returns the value of number of revenue/spend; <br>  effectively <em>(web revenue + app revenue)/(search spend)</em></pre><pre>- Final result is <em>Result(Facebook) + Result (Google Ads),</em> <br>  which is clearly <strong>wrong</strong> as it&#39;s <strong>summing up two different ratios</strong>.</pre><pre><strong>What ideally should happen:<br></strong>- Facebook returns the value of –<br>  <em>web revenue, app revenue and social spend</em><strong> </strong>separately</pre><pre>- Google Ads returns the value of –<br>  <em>web revenue, app revenue and search spend</em><strong> </strong>separately</pre><pre>- Final value i.e. <br>  <em>(web revenue + app revenue)/(search spend + social spend)</em> <br>  is computed in Ruby instead of at DB level.</pre><h4>On to version 3</h4><p>To address this, we designed our current system which uses topological sort of Directed Acyclic Graph to determine which dependent custom metric should be computed first in the hierarchy of nested metrics. In the above example; <strong>web</strong> <strong>revenue, app revenue, search spend and social</strong> <strong>spend</strong> has to be computed first — which goes into DB for the aggregation for each data source and both <strong>revenue</strong> and <strong>spend</strong> must be computed in application code based on the response from aggregate query to DB, before computing RoI <strong>(revenue / spend)</strong> in order to compute the final value.</p><h4>Architecture</h4><p>The custom metrics are not pre-calculated and stored while fetching the reports from API but are computed on-the-fly when the user opens a report which contains the custom metric. The primary reasons for this restriction are,</p><ul><li>The formula of the metrics can be changed at any point.</li><li>The value of a custom metric is dependent on the arbitrary dimension groups &amp; date ranges in the query.</li></ul><p>The cross data source custom metrics needs to follow the topological order in order to get the correct value. The diagram below explains what happens when the user makes a request for a custom metric. Here, we assume that the request contains only one nested custom metric having multiple levels of custom metric dependencies. Each node (A, B, C, D, E) represents a metric and the edge represents dependency of a metric on another.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VJuppTdC4PBOa3jwdqZ7Ow.gif" /><figcaption>Interactive request life-cycle highlighting custom metric computation.</figcaption></figure><p>When a group of custom metrics is requested, a dependency graph (DAG) is built, and the leaf nodes of the graph (Node E in the above image) are the base metrics. Once the graph is built, the leaf nodes (base metrics) are extracted and pushed to the per data source aggregator, which accumulates and returns the metrics grouped by selected dimensions in the given date range. Once the aggregated base metrics from all data sources are computed, they are seeded to topological sort iterator, which starting from the lowest level, computes the dependencies one-by-one before it computes the nested metrics.</p><p>This process assures that all dependencies of a custom metric are computed correctly, ensuring that the result of the last metric in the graph is also correct. However, it has its own limitations in terms of performance (in Ruby) and the operation that can be done on the computed values like cross data source sorting and filtering etc.</p><h4>Performance</h4><p>The performance of the metric computation directly depends on the degree of nesting in the custom metric. The topological sort iterator iterates over each node of the graph. A node of the graph represents either a requested metric, its direct or indirect dependencies, which are other custom metrics or the base metrics. The iterator starts from the base metrics and iterates over each node. The value of the current node in iterator is evaluated from the expression using ‘<a href="https://github.com/rubysolo/dentaku">dentaku</a>’ library. The values of base metrics are seeded into the evaluator initially, used for the evaluation of the first dependent node. The graph is then recursively compressed down to the requested custom metrics by evaluating each node. The evaluator assumes that it has all the values of variables — that are required to evaluate the node — seeded into it so that it can use them for further computations.</p><p>Now, if we take the above example, it has 3 levels of nesting (7 metrics in total). The last level is always base metrics, which form the starting point of the iterator. The base metrics are always computed by the aggregator at DB level, so there’s no computation that the iterator needs to do. The iterator will loop over once for each subsequent metric and evaluate each metric separately in the graph. i.e.</p><p><strong>level-1:</strong> web revenue, app revenue, social spend and search spend; <br><strong>level-2:</strong> revenue, spend; and <br><strong>level-3</strong>: RoI; for each aggregated row.</p><p>If there are 5000 rows, the total number of expression evaluations for this example will be <strong>5000 rows * 7 iterations for metric evaluations per row = 35,000</strong>. In the worst case that we encountered, the total number of expression evaluations per row was around 65 — that means 3,25,000 evaluations (Lot more additional mathematical operation based on the expression internally handled by the dentaku library) per request. Given that it’s pure computation task and no IO operations are involved, there’s no way to run it concurrently because of <a href="https://en.wikipedia.org/wiki/Global_interpreter_lock">GIL</a> in Ruby as only one native thread can execute at a time.</p><p>To overcome this problem, we are exploring different options to move this part of the application. However, GIL continues to be the problem in Ruby-MRI, so we have moved this part to the immediate alternative JRuby in order to avoid rewriting this piece. We also get the advantage of the direct use of third-party Java libraries by using JRuby. Of course, we will face the issues from both Rails and JVM, but that should be manageable for now.</p><h4>In Closing</h4><p>The whole idea of breaking custom metrics into its dependencies and computing dependencies for each data source through DB aggregation, and then iterating over the topological graph to further compute nested metrics is because of the limitations on DB architecture level. We are using MongoDB as OLAP data storage and have separate collections for each data source. As a result, we need to collect the values of base metrics from each collection since sharded MongoDB doesn’t do joins well and we have to take care of the nesting on the application level as described above. We are not storing the data of all sources in a single collection due to operational constraints that are outside of the scope of this article. But we are exploring other solutions which will allow us to overcome these constraints. Our pipelines are read (aggregation) and update heavy, and our aggregation queries to MongoDB sometimes take minutes to respond even in a single channel collection.</p><h4>We Are Hiring!</h4><p>We are building something really exciting and have great customers, investors and lots of unsolved, hard engineering problems. We are hiring across roles in the backend, DB performance, and frontend teams. If you are looking for high-quality work and if you like what we are doing go ahead and <a href="https://clarisights.com/careers/openings">apply here</a>!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=da4a90fcb7ff" width="1" height="1" alt=""><hr><p><a href="https://medium.com/clarisights/resolving-metric-dependency-expression-with-dag-ast-da4a90fcb7ff">Evaluating complex arithmetic expressions at scale</a> was originally published in <a href="https://medium.com/clarisights">Clarisights</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What’s it like to work at Clarisights?]]></title>
            <link>https://medium.com/clarisights/whats-it-like-to-work-at-clarisights-e593e34b801e?source=rss----284fe0b03aa7---4</link>
            <guid isPermaLink="false">https://medium.com/p/e593e34b801e</guid>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[culture]]></category>
            <category><![CDATA[recruiting]]></category>
            <category><![CDATA[startup]]></category>
            <category><![CDATA[hiring]]></category>
            <dc:creator><![CDATA[Srijan]]></dc:creator>
            <pubDate>Fri, 22 Feb 2019 14:51:14 GMT</pubDate>
            <atom:updated>2019-07-25T02:52:07.153Z</atom:updated>
            <content:encoded><![CDATA[<p>March is almost here and 2019 is already two months old. How were your first couple months in this new year, guys? I’m currently in Bengaluru and recently started working with the Growth team at Clarisights. It has been an exciting month of learning, new experiences, and meeting new people!</p><p><em>First things first, what is Clarisights?</em></p><p>Clarisights is a reporting and analytics platform that enables marketers to make sense of and gain unique insights from data. Our platform is a seamless in-browser experience — users can add marketing data sources and begin answering questions on the data, minutes after signing up. Behind the scenes, there’s a lot going on — our backend ingests the data, performs multiple transformations and stores it in our own database. With Clarisights, marketers can build reports to answer questions like <em>“What are my largest sources of traffic and what are the major cost sinkholes?”</em> and then also quickly drill down into granular details like <em>“What kind of ad copy has the least customer acquisition cost”</em> or <em>“Which ad creatives are driving the maximum RoI?”</em> Interesting, isn’t it? Do you want to know the back-story of Clarisights? <a href="https://youtu.be/GFGAcxPragY?t=5919">Check out this video</a> by our CEO, Arun at the Techstars 2018 demo day!</p><p><em>Coming back to me,</em> my first month at Clarisights was dedicated to figuring out how the product works, how we are solving problems for our customers, why our customers chose us and so on. I spoke to everyone including the founders and it was very exhilarating to know how they came up with the idea of Clarisights — heard the company story as they saw it with their own eyes. But, I was able to truly appreciate what an exceptional product we’re building only once I started using it. It was like that “wow” moment you have when you do something new and completely enjoy it — like your first Uber ride or Swiggy delivery. It also allowed me to closely observe the challenging parts of the UX. Looking at the product with fresh eyes was something I could only do once while I was still new, so I tried to make the most of it.</p><p><em>There are few things I like most at Clarisights.</em></p><ol><li><strong>Product</strong> — Clarisights was founded to empower performance marketing and growth teams by creating a platform that gives them insights and superhero powers they have always wanted. Rather than wrestling with spreadsheets and legacy BI platforms that can’t keep pace with their growing volume of marketing channels and data, marketers can use Clarisights to independently explore and analyse all their data in a single and intuitive interface. Marketing teams around the world use Clarisights for every aspect of their performance marketing workflow — and this shows in the level of engagement. On an average, marketers using Clarisights spend over 50 minutes on the platform per day.</li><li><strong>Responsibilities and Ownership </strong>— At Clarisights, you are given freedom to experiment, sense of ownership of your work, license to fail and you have peers who care just as much as you about their work. We have a trust by default policy — i.e. we trust that you will get the work done and won’t micromanage. There’s an immense learning and growth opportunity in small, multi-functional, tightly-knit teams of smart folks here. You want to know, who you would be working with? <a href="https://www.clarisights.com/careers">Check this page out</a>.</li><li><strong>People</strong> — want to know how it’s like to work at Clarisights : flat hierarchy, everyone is friendly, you can crack jokes anywhere anytime, and go to parties with your colleagues. At Clarisights, we live and breathe the company and the work — we are just not a team but a family and it’s a place that makes you feel excited about going to work everyday. When making a decision, we take the holistic approach — everyone’s voice is heard. We are thoughtful about our cross-functional work, customer impact, and each unique touchpoint that goes into the journey of bringing every experience to life. The final product (no matter how big or small) is equally as important as the journey to get there. When decisions are made there is no ultimate decision maker, the title does not matter — whether you are an “Intern”, “Junior Developer”, or “CEO” — your voice carries weight in every discussion.</li></ol><p>Also, we love to play Table Tennis in our free time. Table Tennis is a fierce but sociable competition here. Frequently throughout the day various pairs will take to the table and fight it out to 21-points — before laughing and amicably shaking hands. :D</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fgiphy.com%2Fembed%2Fsz3pnTuOYyupa%2Ftwitter%2Fiframe&amp;display_name=Giphy&amp;url=https%3A%2F%2Fmedia.giphy.com%2Fmedia%2Fsz3pnTuOYyupa%2Fgiphy.gif&amp;image=https%3A%2F%2Fi.giphy.com%2Fmedia%2Fsz3pnTuOYyupa%2F200.gif&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=giphy" width="435" height="355" frameborder="0" scrolling="no"><a href="https://medium.com/media/5b5031bfe292b49a0e8c358d58d2fb50/href">https://medium.com/media/5b5031bfe292b49a0e8c358d58d2fb50/href</a></iframe><h3>Why you should join us</h3><p>I can think of two reasons: a real problem-solving product with a huge addressable market, and you get to work with smart folks who can help you grow and learn. Clarisights is a company that gives you opportunities, recognises your impact and cares about you — you’re not just a resource here, you are a member of the family.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oEQP_ZmVX4wUMCh-DdnW8g.jpeg" /></figure><p>The kind of engineering challenges, we face —</p><ol><li><a href="https://youtu.be/Vo7gUkWSIlc">Ashu Pachauri</a> talking about our backend engineering and infrastructural challenges.</li><li><a href="https://youtu.be/u8Bixn6wCVc">Rohan Malhotra</a> talking about frontend engineering and performance conundrums.</li><li><a href="https://youtu.be/ZzG9QUeB8w8">Yash Arora</a> talking about building a cohesive product experience.</li></ol><p>We have an exciting road ahead, and an early-user group of intelligent marketers from all around the world. They range from EdTech to takeaway brands, and publicly listed multi-billion dollar brands to high growth startups paying us thousands of dollars every month.</p><p>After recently raising a $2.5M round from top European VCs and angels, we’re looking to scale the product both from an engineering and growth perspective. To achieve this, we are building a team of sharp, creative, passionate builders with a “get-shit-done” attitude, the kind of people who see opportunity where others see roadblocks.</p><p>We genuinely love what we do and we’re looking for others who can share the same passion. Here, you will be judged on your work and your aptitude, not just on your pedigree. If this sounds like you, <a href="https://www.clarisights.com/careers/openings">check out our open roles</a>!</p><p>PS - We recently went to our first offsite and if you’d like to see how our culture is, you should surely watch <a href="https://youtu.be/_xBSPLjPR2c">this video</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e593e34b801e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/clarisights/whats-it-like-to-work-at-clarisights-e593e34b801e">What’s it like to work at Clarisights?</a> was originally published in <a href="https://medium.com/clarisights">Clarisights</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The journey of Custom Metrics at Clarisights]]></title>
            <link>https://medium.com/clarisights/the-journey-custom-metrics-at-clarisights-6f747125b03f?source=rss----284fe0b03aa7---4</link>
            <guid isPermaLink="false">https://medium.com/p/6f747125b03f</guid>
            <category><![CDATA[product-development]]></category>
            <category><![CDATA[product]]></category>
            <category><![CDATA[product-management]]></category>
            <category><![CDATA[digital-marketing]]></category>
            <category><![CDATA[product-design]]></category>
            <dc:creator><![CDATA[Pranav Revankar]]></dc:creator>
            <pubDate>Tue, 12 Feb 2019 05:46:34 GMT</pubDate>
            <atom:updated>2019-02-12T05:55:36.885Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/817/1*LD_lj_uzz9WRTJ0TfHkLVA.png" /></figure><p>This article is about how a simple feature spanned its journey to become one of the most used and also complex features of Clarisights. Custom metrics, as the name implies, allows marketers to create their own metrics on Clarisights. For example, to compute the average cost of acquiring a customer, the formula is:</p><pre>Customer Acquisition Cost (CAC) = Cost / Acquisitions</pre><p>Basically, the CAC can be calculated by simply dividing all the <em>costs spent</em> on acquiring more customers (marketing expenses) by the <em>number of customers acquired</em> in the period the money was spent. For example, if a company spent $100 on marketing in a year and acquired 20 customers in the same year, their CAC is $5.00. Looks simple, right? Maybe not!</p><p>When the first version of custom metrics was built, in Jan 2018, it could do the above example. But, with a diverse set of customers coming in, we encountered a diverse set of use cases that they brought along:</p><p><strong>Checkpoint 1: Custom metric from multiple data sources</strong><br>Clarisights provides integration with every major advertising and analytics channels (data sources) and marketers can build a single dashboard of all these data sources. The ability to build custom metrics from multiple data sources is no brainer. One of the most popular cross-channel custom metric is RoAS i.e. Return on Ad Spend (a.k.a ROI) which equals Revenue/Costs. The Revenue is obtained from Google Analytics, CRMs or App Analytics and the Costs from Advertising channels like Facebook, Google Ads etc.</p><p>Return on Ad Spend = Revenue(App)+ Revenue(Web)÷ Total Costs</p><p>Checkpoint reached — March 2018</p><p><strong>Checkpoint 2: Nested Custom metrics</strong><br>For a very long time, we didn’t support nested metrics, i.e. being able to use a custom metric into another custom metric. There was a workaround for this. Say, you want to create a custom metric using RoAS (the custom metric from checkpoint 1). Instead of using RoAS in the formula, you can write it as Revenue/Costs. But soon, the number of nesting started increasing from 2</p><pre><em>custom metric -&gt; custom metric</em></pre><p>to up to more than 5</p><pre><em>custom metric -&gt; custom metric -&gt; custom metric -&gt; custom metric....</em></pre><p>and obviously, the workaround couldn’t work. Fast forward to today, we support any kind of nesting of custom metrics in Clarisights.</p><p>Checkpoint reached — May 2018</p><p><strong>Checkpoint 3: Conditional custom metrics</strong><br>Almost every consumer-facing company (e-commerce, takeaway, edtech, etc) operates both an app and a website. The marketing efforts are divided between these two modes of acquiring new customers. Separate ad campaigns run for promoting the app and the website. In fact, the tools used to measure the performance of these campaigns are different as well. Whereas Google Analytics is the most popular choice for measuring website performance, Appsflyer and Adjust are used to measure the performance of App campaigns. You might be wondering where is this leading to? This led to a new feature addition to the custom metrics — custom metric based on conditions. As the revenue of App campaigns was measured in Adjust/Appsflyer and that of web campaigns on Google Analytics, the formula for Total Revenue looked like this:</p><pre>Total Revenue = <br>IF (Campaign Type=&quot;Web&quot;) THEN GA:Revenue <br>ELSEIF (Campaign Type=&quot;App&quot;) THEN AJ:Revenue</pre><p>Maybe the image below from our product exhibits this better -</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/605/1*voGWxxKoyTGW76FiuEvNPA.png" /></figure><p>(Note: GA is Google Analytics and AJ is Adjust).</p><p>Checkpoint reached — August 2018</p><p><strong>Checkpoint 4: Adding functions in custom metric<br></strong>To calculate the monthly and weekly run-rate and compare it with the targets, we added the functions like {days_remaining_in_month}, {days_so_far_in_week} in custom metrics. Marketers can now check if they are at the right pace to hit the monthly targets.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/439/1*ezlYgo_A8HHcNZBcqFBFGQ.png" /></figure><p>Checkpoint reached — October 2018</p><p><strong>All Checkpoints reached?</strong></p><p>After multiple iterations, we have now added all the above features in custom metrics. But frankly, we don’t know what’s the final checkpoint for custom metrics. And we don’t want to. Its the journey that is more exciting for us and we are sure there are gonna be many more interesting features in the future.</p><p><em>We are currently working on </em><a href="https://en.wikipedia.org/wiki/Weight_function"><em>Weight function</em></a><em> and supporting date range in the custom metrics formula. Believe me, building some of these is super-tricky and we are exploring changing our databases to solve this. At Clarisights, we love challenges, a lot. If you are someone who loves such challenges, I suggest you check out our </em><a href="https://clarisights.com/careers"><em>careers page</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6f747125b03f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/clarisights/the-journey-custom-metrics-at-clarisights-6f747125b03f">The journey of Custom Metrics at Clarisights</a> was originally published in <a href="https://medium.com/clarisights">Clarisights</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Thank You, Early Adopters!]]></title>
            <link>https://medium.com/clarisights/thank-you-early-adopters-3d966ed6125f?source=rss----284fe0b03aa7---4</link>
            <guid isPermaLink="false">https://medium.com/p/3d966ed6125f</guid>
            <category><![CDATA[startup]]></category>
            <category><![CDATA[product-design]]></category>
            <category><![CDATA[customer-service]]></category>
            <category><![CDATA[product-management]]></category>
            <category><![CDATA[customer-experience]]></category>
            <dc:creator><![CDATA[Pranav Revankar]]></dc:creator>
            <pubDate>Wed, 02 Jan 2019 11:18:36 GMT</pubDate>
            <atom:updated>2019-03-13T22:12:02.073Z</atom:updated>
            <content:encoded><![CDATA[<h3>Thank you, Early adopters!</h3><p>I spend my days helping our users navigate through the tad complex product that we are building at Clarisights. My experience over the last year has taught me that if you’re an early adopter, you are destined to encounter challenges. You might get the latest and greatest new OS on your phones before all of your friends, but there’s a good chance that the shiny new features might accidentally stop working leaving you disconnected from the world. Things will crash, capacities will be tested, bugs will surface, and sadness will occur. But if you’re lucky, the fruits of your frustration will eventually flourish into something amazing that your technophobe friends like me will crave while they spin the wheels on their rotary dial phones.</p><p>And yes, we at Clarisights are lucky to have some great early adopters, who are willing to hack, stumble and fight through early release cycles. The reality is that products during the early stages typically are lean. And by lean I mean, “it’s got some holes” (and that is still an understatement). When we launched the first version of Clarisights in October 2017, there were features missing and parts of user experiences were confusing. And yes, there were bugs, a lot of them.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/650/1*VscTS-AfQ6qMtiibfhaitw.jpeg" /></figure><p>Without early adopters, Clarisights might still be wandering around this first version of the dashboard. Looking back at our first versions of our product, I cringe, a happy cringe. I have an appreciation for the hard work and growth accomplished by our team but the real heroes are the users who followed us through, the sometimes frustrating roller-coaster ride of developing a startup.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GZ48HGpA7rkIuIyMcnSaJQ.png" /><figcaption>First version — October 2017</figcaption></figure><p>But thanks to our early adopters, we’ve been able to find and fix a lot of bugs in a very short period of time. We have also improved the UX and added a bunch of features that make data exploration super-easy on Clarisights. In this process, we also rebranded ourselves from AdWyze to Clarisights (our founder, Arun wrote on our <a href="https://clarisights.com/blog/our-new-identity-clarisights">new identity</a>).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Kj1M9ynyDt3h0mCUYZQieg.png" /><figcaption>Clarisights in October 2018</figcaption></figure><p>The first set of Clarisights’ customers will always have a special place in my heart. I’ve spent the majority of my waking (and sometimes dreaming) hours ensuring that they are having a pleasant experience on Clarisights. Believe it or not, it’s no simple task to build a cross-platform reporting platform that automatically fetches millions of data points from different sources and collates them onto a simple and intuitive dashboard. But that’s what we’re doing well today, thanks in large part to the powerful feedback we received from early adopters. And we’re going to get even better.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/250/1*MUdSViV2DfI-HJFxEDKLZQ.gif" /></figure><p>Their patience through the hiccups — and let’s be honest, some of the hiccups may have felt like head-smashed-on-the-keyboard moments of omg-why-didn’t-we-realize-that — means a lot to us. And while our users have been ever-so-patient, we’ve worked hard to improve every aspect of their experience on Clarisights.</p><p>And this isn’t lip service. We’re a small but growing team with huge ambition and long to-do lists. We’re moving fast, trying new things, and thriving on the goal of democratizing data for the marketing teams. But one thing that won’t ever change is how much we love and appreciate our customers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FugD0FypTuosNudXw4qCYw.jpeg" /></figure><p><em>This article was </em><a href="https://clarisights.com/blog/thank-you-early-adopters/?utm_source=medium&amp;utm_medium=blog&amp;utm_campaign=thank-you-early-adopters"><em>originally posted</em></a><em> on the </em><a href="https://clarisights.com/?utm_source=medium&amp;utm_medium=blog&amp;utm_campaign=thank-you-early-adopters"><em>Clarisights</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3d966ed6125f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/clarisights/thank-you-early-adopters-3d966ed6125f">Thank You, Early Adopters!</a> was originally published in <a href="https://medium.com/clarisights">Clarisights</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[My PM summer internship at Clarisights]]></title>
            <link>https://medium.com/clarisights/my-pm-summer-internship-at-clarisights-bb2deeea8de4?source=rss----284fe0b03aa7---4</link>
            <guid isPermaLink="false">https://medium.com/p/bb2deeea8de4</guid>
            <category><![CDATA[product-management]]></category>
            <category><![CDATA[internships]]></category>
            <category><![CDATA[startup]]></category>
            <category><![CDATA[marketing]]></category>
            <category><![CDATA[summer-internships]]></category>
            <dc:creator><![CDATA[Anmol Chaman]]></dc:creator>
            <pubDate>Mon, 16 Jul 2018 12:50:20 GMT</pubDate>
            <atom:updated>2018-07-16T13:25:44.573Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/906/1*4ZcvY3BvDMoTt_ei2j9YyA.jpeg" /></figure><p>An introduction right? So that’s how it begins. I introduce to evince my credibility so that you don’t junk your time here, getting to know the experience of an intern at a startup. Well, I won’t do that. Trust me and read along, you might find my experience interesting.</p><p>I write this as my internship at Clarisights is about to conclude. I learned a lot in these 45 days in Bangalore. Be it how to manage time, juggling work from the internship, my startup in Kanpur, Vox populi or Senate, to being a better product manager, understanding the value of money and starting valuing my relationship more. This has been a great experience in terms of the work, the weather this city has, and the people I befriended here. And bless the multicultural nature of our nation, it’s not easy to maintain sanity and calm in a place where any of the three languages you know aren’t understood.</p><p>Now that you have read this far, I’ll begin first with introducing myself and how I landed in an internship here.</p><p>I am pursuing major in Aerospace Engineering from IIT Kanpur. And like most of the engineers, I won’t be pursuing that after graduation. I started up a few months back, a cab aggregator platform BookKar IITK. Working on it made me a product guy at heart and I thought it was time to see how work is as a Product Manager. So I started looking for internship opportunities in this area.</p><p>So the next thing I know, is after morning lectures and startup work, I had a profile on Angelist ready and active to look for an internship. I wanted to intern at a start up because working in it would give me an experience of how fast scaling startups work. It could give me an experience that I would use to make good decisions for my own. Those crazy startup fanatics we see in movies, I wanted to see them in real life.</p><p>I got in talks with a few startups and found myself worthy of the opportunity but soon a fear of responsibility disguised as self-confidence kicked in. I started growing apprehensions about those internships. Sometimes when things seem difficult or uncertain, we try to replace those feelings with a developed confidence for something less important and easy to manage. I now feel, the fact that I had to work in a company in a new city was intimidating and in contrast spending an unproductive summers in college was easier. Hence, I started feeling guilty for not working on my startup (knowing the work could be easily handled remotely), rather developing a product for someone else.</p><p>I said no to a few startups and booked a flight to Kanpur to “Save my startup” which wasn’t even an inch below water. *ping!* and I received a mail from Clarisights, asking for an interview. Thank God my parents and brother told me to get my shit together and go for it as this was what I always wanted in the first place. A scaling startup with founders who are leaders in their areas and the most important, a great product. Clarisights is the decision platform for growth. The team is building a real-time, interactive and contextual reporting platform for performance marketing teams.</p><p>Now that I bid adieu to my internship, I am proud to say that I am leaving Bangalore with more knowledge about the product space, startup ecosystem and with six kgs less than what I was (yes I joined a gym too).</p><p>It was my first time as an in-office intern. All my confidence was switched off on day-one at work. The product intimidated me as it was complicated, and I didn’t have much of a marketing background to understand it completely. I was the youngest there and that kind of intimidated me more. However, as days passed I began settling in. The people in the office were very welcoming, and it won’t be overstating that I will miss going there. <a href="https://www.linkedin.com/in/pranavrevankar/">Pranav</a>, Product thinker is a great product guy. He was helpful throughout, gave me due and sometimes even more credit than what I deserved. A true gentleman he is. I observed how he managed all the tasks at hand and still was always available when I needed help. <a href="https://www.linkedin.com/in/sumangalv/">Sumangal</a>, growth and sales guy is a genius at what he does. Also he has a vast knowledge of online products. You want to know about any product out there, ask him and he’ll pitch it so well that you’ll want to buy it instantly :P. He was very sweet and helpful throughout. <a href="https://www.linkedin.com/in/ankur-gupta-3a24438/">Ankur</a>, co-founder Clarisights is a great engineer. I remember preparing extensively before giving him a presentation as he used to think about the scenarios we had not thought about and problems that made our designs not relevant anymore. It prepared me to think the unthinkable and do the undoable.</p><p>I accomplished designing some product features like data currency conversion (for multinational enterprises) and an alert feature for marketers to know when a particular KPI in their data hits a limit. I built an online task database for all engineers in team and ensured it’s maintenance for better effort distribution and team management. In addition to the platform, product and team management, what I learned a lot is how marketers look at data. What are the metrics and parameters they value and how they implement their digital marketing campaigns. From looking at the marketing stack of enterprises to startups, I understood the correlation between the size of a company, the type of marketing campaigns and the relative importance of various marketing KPIs. Since the platform helps marketers analyze data, I have started valuing data-driven decision making more, which I plan to make inherent in my startup.</p><p>As my last day at Clarisights is here and I’ll be leaving for campus to begin my tenure as Acting President, Students’ Gymkhana for some time, I think being here for an internship has been a great decision that has made me more passionate about working on my startup and the experience I got here will definitely help me to make a great product one day.</p><p>So long!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bb2deeea8de4" width="1" height="1" alt=""><hr><p><a href="https://medium.com/clarisights/my-pm-summer-internship-at-clarisights-bb2deeea8de4">My PM summer internship at Clarisights</a> was originally published in <a href="https://medium.com/clarisights">Clarisights</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Scaling Backend 100X]]></title>
            <link>https://medium.com/clarisights/scaling-backend-100x-d2da0a4dd102?source=rss----284fe0b03aa7---4</link>
            <guid isPermaLink="false">https://medium.com/p/d2da0a4dd102</guid>
            <category><![CDATA[performance]]></category>
            <category><![CDATA[ruby]]></category>
            <category><![CDATA[scaling]]></category>
            <category><![CDATA[engineering]]></category>
            <category><![CDATA[ruby-on-rails]]></category>
            <dc:creator><![CDATA[Ankur Gupta]]></dc:creator>
            <pubDate>Tue, 12 Jun 2018 05:15:01 GMT</pubDate>
            <atom:updated>2018-10-24T08:51:26.670Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>Introduction</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NT_T9udwCZwHTWBvCNFXfQ.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/photos/Ro6oAfHPc9A?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Samuel Zeller</a> on <a href="https://unsplash.com/search/photos/scale?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>The post will show how we scaled a lot of our time consuming background jobs 100X and more. Obviously, 100X is relative to where we were before. Our previous <a href="https://blog.adwyze.com/how-we-scaled-our-rails-backend-135d14a165ca">post</a> covered some of the optimizations we did earlier and how the setup looks like. For sake of brevity, we won’t repeat that here but briefly mention the setup and db sizes as of today. We pull data (primarily time series) from hundreds of external sources and store it in Postgres. Since our queries are analytic in nature and require heavy joins, we do roll ups and precompute those joins and store the denormalized data in MongoDB. Since our <a href="https://blog.adwyze.com/how-we-scaled-our-rails-backend-135d14a165ca">previous post</a> on scale, our Postgres has grown from ~100GB to ~1TB and Mongo from ~40GB to ~160GB. We face scale and performance issues at all stages of our ETL pipeline:</p><ul><li>Ingestion: While ingesting external data into PG,</li><li>Precomp: While computing the rollups data (at daily and hourly level) and pushing to MongoDB</li><li>Querying: While serving our analytics queries</li></ul><p>In early January this year, my co-founder got us a new customer who was interested in our product. They are in Europe and a $6 Billion dollar company. This is our first unicorn customer. While we all were super excited to get them on board, our fear was whether our backend will be able to handle their scale. Me and my co-founder did a quick back of the envelope calculation. Based on that we concluded that our new customer is <strong>more than 100 times</strong> <strong>bigger</strong> than all our existing customers combined! We were jumping in excitement at the prospect of getting such a big customer but the problem was that our backend was no way ready to handle this scale. Being an enterprise product, sales cycle are longish (3–6 months), so thankfully we had time before the customer would come onboard fully. We jumped straight to scaling our backend. Goal was clear: we need to scale at least 100X!</p><h3><strong>What to scale?</strong></h3><p>As mentioned earlier, we needed to scale all our stages — ingestion, precomp, and querying. All our background jobs are either in ingestion or precomp phase and querying phase is simply optimizing our Mongo queries. We are big fan of instrumentation &amp; monitoring and use Datadog and InfluxDB/Grafana to track job run times and query latencies. That served as a good starting point. All jobs that are taking too long to run need to be optimized. Lets start with the ingestion pipeline where we read external data, check whats in our PG and update the data.</p><h3>How to scale? — detour</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KEZ54sMi3slhkbVaSJ6pGw.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/photos/TrqQBU3xMEs?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Kees Streefkerk</a> on <a href="https://unsplash.com/search/photos/detour?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>The first thought was Ruby is slow and MRI implementation in particular is not geared for true parallelism. In particular, because of <a href="https://wiki.python.org/moin/GlobalInterpreterLock">GIL</a>, multiple threads in a Ruby process cannot be scheduled on multiple cores — they all run on the single core on which the process was started. JRuby does not have this limitation and therefore, that seemed a good candidate and standard benchmarks show that it is 3–5X faster than MRI. We then stumbled upon <a href="https://github.com/oracle/truffleruby">Truffle</a>, which is another implementation of Ruby language and <a href="https://pragtob.wordpress.com/2017/01/24/benchmarking-a-go-ai-in-ruby-cruby-vs-rubinius-vs-jruby-vs-truffle-a-year-later/">benchmarks</a> show it to be 30X faster than MRI. However, Truffle was quickly ruled out as it is still in active development and is missing key network implementations (like nokogiri gem) that makes it unusable with Rails. So, JRuby, it is, right? But wait, lets take a step back and understand what are we trying to achieve here. Didn’t we answer that question already? We need to scale 100X, period. Still, It is important to go back and answer <strong>scale what</strong>? A typical web application involves a DB. And in our case, we have two DBs! CPU is almost never the bottleneck — it is almost always the I/O. Too many DB queries (especially in a loop), waiting for an external service to respond, network calls, reading/writing from/to disk and so on. So even if we make Ruby “faster”, it won’t help in our case (we are a data heavy company). The benchmarks are not misleading — they are typically for CPU intensive work loads and it is important to understand that before jumping the gun. Also, moving from MRI to JRuby is a non-trivial migration and it was clear that gains will be limited. Hence, we ditched changing our run time environment for Ruby.</p><p><em>A faster Ruby is not going to magically fix a “bubble sort” hiding deep in your code</em> — Sam Saffron in a <a href="https://samsaffron.com/archive/2018/06/01/an-analysis-of-memory-bloat-in-active-record-5-2">recent post</a></p><p>Throwing more machines was the other obvious solution. Apart from being an expensive affair, we strongly felt that it was premature as we ought to understand what is causing the slowness in the first place. Therefore, it was time to become<em> morbidly obsessive</em> at taming our background jobs by finding inefficiencies and nuking them.</p><h3>How to scale? — again</h3><p>This section will cover a variety of techniques we applied to identify and remove the bottlenecks. Most of them are agnostic to Ruby on Rails and can be applied to other platforms.</p><blockquote>Be extremely lazy</blockquote><p>As alluded in previous section, our bottlenecks were I/O. Lets focus on the ingestion pipeline where we need to fetch data from external APIs, compare the external data with our PG and update PG. Ingestion itself can be broken down into two parts: 1) Fetching ad objects and their attributes and dimensions, 2) Fetching the ad objects stats, also known as the report jobs. For channels like AdWords, number of ad and keyword objects can be huge — 10k-100k per customer. Fetching these many objects say every hour for hundreds of customers can’t scale. We noted that, practically, in any time delta period, only a fraction of objects change. AdWords has the wonderful CustomSyncService API, which tells the change since a given time. We started querying that first to get list of campaigns, adgroups, and ads that have changed and refresh only those. That gave a huge performance boost as instead of fetching 100% objects, we were fetching only changed objects which typically would not be more than 1% of total data. 100X gain right there! Moral here — be lazy, we mean, really lazy and fetch only that you absolutely need.</p><blockquote>Be smart, challenge status quo, don’t hesitate to undo things</blockquote><p>Now switching to reports. AdWords stats data change for up to 30 days. So even if we have fetched data in past, each day, we need to keep fetching data for the last 30 days. We call it our backfill job. Our backfill job was giving big headaches as it was supposed to run once every 12 hours and a single run itself was taking 1.5 days (soon after the new unicorn customer added a handful of their accounts). Yes, you heard it right, 1.5 days to fetch all the data. How in the world are we gonna solve it as we were already doing batch reads/writes, running multiple threads — the <em>obvious</em> parallelism had already been implemented. We then looked at what it was doing. It was fetching data from AdWords, computing the delta between what is between AdWords and our PG and then writing that to PG. During delta computation, we had a sprinkling logic that would uniformly sprinkle the delta data over all rows for any particular day. Consider it legacy code and an approximation to hide all delta being visible in a particular hour. The problem was that for each object and each day, it had to look at least 24 rows and write back the delta to them. So 24 reads and 24 writes. Since anyway it was an approximation and goal of backfill was more to make sure numbers match at day level and not necessarily at each hour level, we got rid of it and dumped all the delta in the 23rd hour. So now instead of 24 reads and 24 writes, for each object, a single read and write for each day. This single change alone brought down the time from more than 120k seconds to less than 5k seconds (a day has 86.4k seconds). A full 24X gain! Lessons here: challenge status quo, understand your DB queries — especially the ones being done in a loop (or nested loop like ours) and see which one of those loop DB queries you can get rid of.</p><blockquote>Index in past != index in future</blockquote><p>In another instance, we were pulling data from Google Analytics and the job was taking forever to finish even for fetching only last 2 days of stats. It was supposed to run every hour and it was taking 178k seconds (~2 days) in worst case. We nailed it to a missing index — even though we had indexes in place, over time the code changed and a new query was no longer using the old index. The old query had columns (A, B, C, D) in its where clause and we had index on (A, B, D). The new query had (A, B, C, E) in its where clause. We removed the old index and added the common prefix index (A, B, C). Since cardinality of D was more than that of C, by getting rid of the old index, we actually brought down the index size and were able to use it for both types of queries. This simple change brought down the run time to about 1k seconds — a cool 170X+ gain! While tools like PgHero can suggest you missing indexes, it is ultimately to the application builder to really understand what their code is doing and what is the right index to use.</p><blockquote>Reduce db ops, bulk writes</blockquote><p>One of our data decoration job was taking forever to finish — won’t finish even in 24 hours. The decoration involves running multiple annotators on each ad object and writing the annotations to PG and Mongo. For simplicity, lets say we have about 10 annotators that need to run on each object. We had the obvious optimizations in place like only run annotators for recently changed objects. The original implementation would run each annotator, generate the mongo op that will update the annotator hash that we had in the Mongo document for the object. The hash in Mongo document looks like tags: {&lt;tag name&gt;: &lt;tag value&gt;} where each annotator writes a single entry (key/value pair) in the hash. The annotators were multi-threaded as clearly there is lot of I/O involved. Yet the annotator job will take really really long time to finish (wouldn’t finish even a day). Because the annotator involved reading from PG, writing back to PG and then pushing the changes to Mongo, we thought batching the reads and writes will help. So we added batched writes both to PG as well as Mongo. Yet this didn’t help — our job will continue running for more than a day and no hope of finishing! We took a more closer look at the logs and tried identifying where it was getting stuck. Our analysis showed that it was the Mongo writes. The next question was how can we reduce the number of ops we generate. That immediately gave the idea that we do not need to generate a mongo write for each annotation to each object (corresponds to one mongo document). Instead, run all annotators on the object, write all the changes to PG and then have a single bigger update that updates all the key/value pairs in the hash in Mongo. We also instrumented the total number of ops generated (rather naively by having a log statement showing the number). In our test setup itself we saw that in a test where earlier it was generating more than 500 ops, after the simple move, the ops reduced to about 50. After this optimization launched, the job that wasn’t finishing at all finished in 10 minutes! Since we did lot of other batch write optimizations too, it is hard to isolate the exact impact of reducing the mongo ops, but we estimate that overall, we had at least 144X gain (remember the job was not finishing even in a day).</p><p>In yet another instance, one of our external API jobs was taking more than a day to finish. We were pulling last 7 days of data and the timeseries file was a JSON file as big as 100MB. The job had to parse the file, accumulate stats in-memory by a group-by key and then write that to DB. The stats are hierarchical in nature — campaigns contains adgroups and adroups contain ads and each ad have some 20 metrics for each day and OS type. We were already doing batch writes but in order to keep the code simple, we were only batching writes within an adgroup. Using <a href="https://blog.adwyze.com/ruby-hotspots-finding-avenues-for-memoization-and-caching-492c6666807a">stackdump</a> from the live job, we identified that the writes to PG were the bottleneck. We kept wondering why they are taking time as we were already batching. It then occurred to us that each campaign has about 5–6 adgroups but each adgroup has only about 1.2 ads on an average. So even though we were batching writes, the earlier batching was poor. We then started aggressively batching across adgroups and even across campaigns and job became at least 5X faster. It still takes 20k seconds to finish — is supposed to run once per day, so kind of ok for now! Interestingly, when it was taking more than a day to finish, it was also hogging as much as 40GB of memory. Now that it has become faster, memory does not peak beyond 5GB.</p><blockquote>Hash With Indifferent Access (HWIA) is a murderer</blockquote><p>The tag annotator job that we scaled 144X by reducing Mongo ops a month later again became dog slow (about 15X slow). We were really shocked to see that as all the hard work to tame it seemed to have gone wasted. After carefully looking at all commits, we isolated it to a change that was done with good intention to detect ad objects that have not changed and hence would not require re-annotation. Our initial check was comparing a JSON hash with a symbolized keys Hash. Those who know Ruby, will immediately spot that JSON hash will have string keys whereas we were comparing it against a symbolized Hash, so even if they are identical except for key type, it wont match. As a result we were unnecessarily re-annotating unchanged objects. One of our engineers spotted that and fixed it in a seemingly innocuous pull request by making the comparison to compare two <a href="http://api.rubyonrails.org/v4.2/classes/ActiveSupport/HashWithIndifferentAccess.html">HWIA</a> hashes instead. While the code was correct, it undid all the gains we made by reducing the Mongo ops. We immediately reverted the change being aware that the change was actually <em>fixing</em> an inefficiency. Post revert, the job was back to its original performant state. Till that time we were liberal with our use of HWIA. Post that <em>traumatic</em> event, we added a check in Rubocop and overcommit config to prohibit use of HWIA and religiously went through the code to exterminate HWIA from our code. We still have few places using HWIA but they have been isolated and measures are in place to make sure they don’t multiply! As an aside, 3 months later we fixed the inefficiency correctly by making the check between two symbolized Hashes. That gave another 10X boost for a combined staggering gain of 1440X (Mongo ops optimization plus the bug fix to correctly compare two symbolized Hashes)!</p><blockquote>Memoization and caching</blockquote><p>We previously mentioned how we got rid of some DB writes and it dramatically improved the performance. It is not always possible to get rid of DB queries. However, it often happens that we are doing the same query again and again in a loop. Note that it is possible that the loop may be <em>invisible</em> i.e. some method may be calling another method and that other method might be executing same query based on the passed arguments. Our CTO, <a href="https://medium.com/u/78d0c2ae2787">Ashu Pachauri</a>, made a <a href="https://blog.adwyze.com/ruby-hotspots-finding-avenues-for-memoization-and-caching-492c6666807a">cool observation</a> that we can use SimpleCov hits/line counter to find lines which are repeatedly being called. That turned out to be an awesome insight — we found many places where we could remember the result of DB query either in local memory or storing it in our distributed Redis cache. We identified <strong>more than twenty</strong> such hotspots. While we don’t have individual improvement numbers, our test suite which used to take as much as 2<strong>0 minutes</strong>, now takes less than <strong>7 minutes </strong>(there were some other optimizations too that we did to make our test pipeline faster, reserving that for a future post). That is a huge huge improvement as we already were using Parallel RSpec. For the record, we run our test job on Jenkins with 8 cores, have 670 spec files, 1.7k <em>it</em> blocks and 17k <em>expects </em>and our test coverage is about 75%. Our tests are more like module test rather than unittest and as such rely on Postgres and Mongo read/writes. We do use FactoryBot and can’t rely on its <a href="https://robots.thoughtbot.com/speed-up-tests-by-selectively-avoiding-factory-girl">in-memory <em>build</em> constructs</a> alone, we need to use<em> create</em> so that our module tests can verify the state of db and do further processing based on that. Note that the memoizations and caching improvements led to significant improvements in production too — for us being able to bring down our test run time numbers seemed even bigger achievement as it improved developer productivity — pull requests could be iterated on 3X faster now!</p><blockquote>Other optimizations</blockquote><p>Will briefly mention some of the other optimizations we did. Each has a reference to external source outlining why it makes sense to do that.</p><ul><li><a href="https://www.mikeperham.com/2018/02/28/ruby-optimization-with-one-magic-comment/">Freeze</a> all your strings with a magic comment — We wrote a handy script to do it in all 3500+ files across our code base.</li><li>Replaced ActiveSupport::Cache with <a href="https://github.com/sorentwo/readthis">Readthis</a> as we rely on Redis heavily for caching — we have <a href="http://www.mikeperham.com/2015/09/24/storing-data-with-redis/">100 databases</a> in Redis with over 13M keys.</li><li><a href="https://www.mutuallyhuman.com/blog/2016/06/28/importing-data-quickly-in-ruby-on-rails-applications">Skip validation</a> during bulk import when using <a href="https://github.com/zdennis/activerecord-import">ActiveRecord-import</a> gem — WARNING: do it only when you understand what you are doing.</li></ul><h3>Query optimizations</h3><p>So far, all the optimizations were geared towards scaling our background jobs. We also needed to make our real-time queries to Mongo faster. Apart from caching query results in Redis, we noticed that our time series data is sparse — many metrics have zero value and yet we were storing them in Mongo. This was a problem as our documents were becoming bigger and bigger with each day (day level rollups are stored in an array in each Mongo document). A quick calculation showed that by not storing zero metrics in Mongo, we could reduce our document size by as much as 60%. We did exactly that and across our 200+ Mongo collections we were able to reduce document size from 15%-60%. This was a tremendous gain as Mongo could retrieve the documents much faster. Beyond this, it was clear that our design of storing time series stats of an object in an array in the Mongo object will lead to document growing indefinitely and making our queries slower and slower each day. Further, Mongo has a <a href="https://docs.mongodb.com/manual/reference/limits/">hard limit</a> of 16MB for document size. Hence, we can’t allow the doc to grow arbitrarily. This led to the sharding project where each object gets sharded — each shard containing only 6 months of data. This was a big change and a future post will outline how we did it and its implications.</p><h3>The Impact</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4xhZAAIA1VW5sDi9PVb9dQ.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/photos/cGwfkwHmt98?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Joshua Earle</a> on <a href="https://unsplash.com/search/photos/impact?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>In hindsight, the optimizations outlined above may look simple but it took more than 5 months to really scale our backend. The optimizations were not easy to identify and without tools like Datadog where we track per job run times, RubyProf, <a href="https://blog.adwyze.com/ruby-hotspots-finding-avenues-for-memoization-and-caching-492c6666807a">stackdump</a>, and SimpleCov, it would have been impossible. Super proud with the scale we have achieved in the past 5 months. While we can present endless graphs for each of the individual optimizations we did, we will present a couple of tables — how the run times looked for the worst jobs in January, in May and now (in June). The numbers clearly show how we were able to dramatically improve our bottom line across the board and not just a one off gain limited to a few jobs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yS-wmaOMOWE4nI4oo2jQMg.png" /><figcaption>Worst 50 job run time before optimizations [January 2018] — does not cover jobs that were taking forever to finish as timing metrics are reported only after they finish</figcaption></figure><p>The above graph only shows the state of affairs in the month of January. Once the unicorn customers channel accounts were integrated (in Feb and beyond), some new jobs started taking days to finish. That is not captured above because Datadog barfs out when we query job run times for longer periods than a month. In May, when we were still in the middle of optimizations, worst job times looked like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*y_1uw7k6cmYr8KikkJFslg.png" /><figcaption>Max job times in [May 2018] — reduced jobs count to show some of the worst performers!</figcaption></figure><p>After all the optimizations, our current worst case times looks like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-_3Pb8YhA59GKXOZ1fguyg.png" /><figcaption>Worst 50 job run times after alloptimizations [June 2018]</figcaption></figure><h3>The Road Ahead</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ijyxy3W1adE3dpfnU4znoA.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/photos/LPffuc1TESQ?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Lance Asper</a> on <a href="https://unsplash.com/search/photos/impact?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>While we are super proud of what we have achieved so far and thankful to our unicorn customer for pushing us out of our comfort zone, we must admit our setup is still relatively small — we still are talking about few terabytes of data. As more customers come onboard, the current choice of MongoDB as our OLAP is not sustainable. There are better alternatives belonging to the class of columnar DBs like Druid, Clickhouse, and BigQuery that are much better suited for analytics queries and can easily scale to peta bytes of data and beyond. Would love to brainstorm with folks who have worked with these DBs and/or otherwise have scaled their analytics pipeline to petabytes of data! I can be reached at ankur [at] clarisights.com. Happy scaling!</p><p>Ciao.</p><p>PS: If you want your Mondays to be awesome, want to be in a state where you look forward to coming to office each day, and want to solve exciting problems like above then <a href="https://angel.co/clarisights/jobs">apply here</a> to join our team!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d2da0a4dd102" width="1" height="1" alt=""><hr><p><a href="https://medium.com/clarisights/scaling-backend-100x-d2da0a4dd102">Scaling Backend 100X</a> was originally published in <a href="https://medium.com/clarisights">Clarisights</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Postgresql — mysterious bad connections issue post Rails 5.2]]></title>
            <link>https://medium.com/clarisights/postgresql-mysterious-bad-connections-issue-post-rails-5-2-c1fbc1e49cbe?source=rss----284fe0b03aa7---4</link>
            <guid isPermaLink="false">https://medium.com/p/c1fbc1e49cbe</guid>
            <category><![CDATA[engineering]]></category>
            <category><![CDATA[monitoring]]></category>
            <category><![CDATA[scaling]]></category>
            <category><![CDATA[ruby-on-rails]]></category>
            <category><![CDATA[postgresql]]></category>
            <dc:creator><![CDATA[Ankur Gupta]]></dc:creator>
            <pubDate>Tue, 12 Jun 2018 05:04:37 GMT</pubDate>
            <atom:updated>2018-06-12T05:04:43.523Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>The Problem</strong></p><p>We have a Rails backend and use Postgresql as our OLTP db. About a month back, we moved to Rails 5.2. Ever since we noticed that our queries to PG in our background jobs will randomly fail. Errors would look like PG::ConnectionBad: PQconsumeInput(). These errors in particular used to come when we used to do find_each on large tables. So we thought may be queries are taking time and we need to optimize them. But then the error started appearing randomly in almost every job regardless of table size. This is when we realized that it has nothing to do with our background jobs but more of a server issue. Since we have thousands of jobs running through out the day, we use PGBouncer to manage the number of connections we open to PG. In pre-Rails 5.2 world, we were using transaction based pooling in PGBouncer and our max connections to PG was a mere 25. Hence we never had this problem. But after Rails 5.2, we were forced to move to session pooling as 5.2 started throwing errors and our migrations were <a href="https://github.com/rails/rails/issues/32622">failing</a>. In session pooling mode, PGBouncer needs to open lot more connections to PG than before. Hence we changed the max_connections in postgresql.conf to 1000 post Rails 5.2 upgrade. Things were fine initially but as mentioned earlier, we started seeing PG connection bad issue repeatedly across all our jobs.</p><p><strong>Failed attempts at diagnosis</strong></p><p>The mystery was deepening and <a href="https://stackoverflow.com/search?q=PG%3A%3AConnectionBad%3A+PQconsumeInput%28%29">Stackoverflow</a>, <a href="https://www.google.co.in/search?q=PG%3A%3AConnectionBad%3A+PQconsumeInput()&amp;oq=PG%3A%3AConnectionBad%3A+PQconsumeInput()&amp;aqs=chrome..69i57j69i58j69i61l3j69i59.1024j0j7&amp;sourceid=chrome&amp;ie=UTF-8">Google</a>, Github<a href="https://github.com/rails/rails/issues/24041"> issues</a> were not turning up a concrete answer to the problem that we were facing. In a few places, we were missing indexes, so we optimized the queries by adding proper index. Still the problem won’t go away. We then tried adding retry logic around some of our batch paginated calls — it helped a bit but it was irritable that we had to put begin/rescue around each of the query blocks to catch exceptions and we were unwilling to accept that as a solution as it was more of a hack than really understanding what was going on.</p><p><strong>The Voila Moment</strong></p><p>We looked at various monitoring graphs in Grafana where we track PGBouncer and PG health. We stumbled upon two graphs that kind of nailed the culprit. The number of client connections to PGBouncer and number of active connections to PG from PGBouncer. We noticed while the former was crossing 1k, the latter seem to have a hard stop at 1k and would never cross it. Bingo! We knew something is wrong here and we found max_connections in postgresql.conf was set to 1k. Of course one month had passed since we had set the max_connections in postgresql.conf to 1k and we had completely forgotten about it!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NtP9C6y4x6U1Bo5a1WRpFA.png" /><figcaption>Number of client connections to PGBouncer — was crossing 1k periodically through last 7 days</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qQSmqJGFGi7085V0l9EMiQ.png" /><figcaption>Number of active connections to PG from PGBouncer — notice how it was <strong>never</strong> crossing 1k till 23rd May 2018</figcaption></figure><p><strong>The Solution</strong></p><p>Its kinda obvious what we had to do. Just increase the max_connections in postgresql.conf to a higher number (we chose to set it to 3k, the same value that we have set in PGBouncer for max client connections). You will notice that after 23rd May evening, active server connections started going beyond 1k (we added the fix around that time) and our Sentry which was flooded with PG bad connection errors suddenly fell silent! Note that while the solution was trivial, from the errors we saw, it wasn’t obvious that PG was dropping our connections because of the max limit.</p><p><strong>Parting Thoughts — Monitoring is important!</strong></p><p>Debugging this would have been a nightmare if we didn’t have the health graphs — in particular the connections count. So highly encourage to have enough monitoring in place for your services unless your services never fail and you know they will never fail :)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c1fbc1e49cbe" width="1" height="1" alt=""><hr><p><a href="https://medium.com/clarisights/postgresql-mysterious-bad-connections-issue-post-rails-5-2-c1fbc1e49cbe">Postgresql — mysterious bad connections issue post Rails 5.2</a> was originally published in <a href="https://medium.com/clarisights">Clarisights</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Monitoring and Reporting Errors from Sidekiq Jobs]]></title>
            <link>https://medium.com/clarisights/monitoring-and-reporting-errors-from-sidekiq-jobs-4b946036c9f4?source=rss----284fe0b03aa7---4</link>
            <guid isPermaLink="false">https://medium.com/p/4b946036c9f4</guid>
            <category><![CDATA[sidekiq]]></category>
            <category><![CDATA[ruby-on-rails]]></category>
            <category><![CDATA[sentry]]></category>
            <category><![CDATA[engineering]]></category>
            <category><![CDATA[error-handling]]></category>
            <dc:creator><![CDATA[Suraj Nath]]></dc:creator>
            <pubDate>Mon, 11 Jun 2018 14:08:16 GMT</pubDate>
            <atom:updated>2018-06-11T14:10:02.004Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nSzDe1v_8LNYr9z3iT1FTA.png" /><figcaption>Configuration to Register Sidekiq Sentry Middleware</figcaption></figure><p>Sidekiq is famous <em>(in the Ruby world)</em> as an easy and simple background job processing solution. Sidekiq covers almost all the use cases that you expect from a background job processing solution but sometimes you need something that is not supported out of the box. That’s when Sidekiq middleware comes in the picture, middleware can extend Sidekiq to run your code before and during job life cycle.</p><p>We, at <a href="http://www.clarisights.com/">Clarisights</a>, use Sidekiq extensively for background jobs. Just like any other piece of software, sometimes background jobs fail. When that happens we want to know why and how that happened.</p><h4>Why do we need this?</h4><p>We wanted to capture every unhandled error from background jobs. We collect all errors in our central error reporting tool (Sentry). Our application is configured to use sentry but we were only getting errors when all retries for a job were exhausted. It was happening because Sidekiq captures any exception occurred in a job (that’s how it decides that a job failed) and uses it to retry failed jobs. It doesn’t raise after capture unless retries are exhausted which will happen after a lot of retries (you can <a href="https://github.com/mperham/sidekiq/wiki/Error-Handling#automatic-job-retry">configure retry</a> policy).</p><h4>Sidekiq Sentry Middleware</h4><p>Sidekiq has notion of <strong>server-side</strong> and <strong>client-side</strong> middleware, Server-side middleware runs around job life cycle while client-side middleware runs before pushing a job to Redis.</p><p>We wrote this <strong>server-side middleware</strong> to report <em>unhandled</em> errors from jobs. If a job fails, our middleware will capture an <em>unhandled exception</em> and report that to Sentry (<em>can be easily adapted to report to other sources as well</em>). After reporting, we will raise that exception so Sidekiq &amp; other middlewares down the chain know that the job failed.</p><p>Raising captured exception again is important here because, if you don’t raise, Sidekiq or any other middleware down the chain will think that job completed successfully.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ba9c8866e436d56fc4f0f3ee26334f90/href">https://medium.com/media/ba9c8866e436d56fc4f0f3ee26334f90/href</a></iframe><h4>How to use it</h4><p>Add it in Sidekiq middleware folder and register it in Sidekiq configuration.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8fe741c9c87355fbb0efa4bce4fbfd68/href">https://medium.com/media/8fe741c9c87355fbb0efa4bce4fbfd68/href</a></iframe><p>It should work out of the box, provided you have Sentry (or any other source where you want to report) configured in your application.</p><h4>Final Thoughts</h4><p>After we started using Sidekiq error reporting, we were able to minimize the impact of failing background jobs. Now we have more visibility into our background jobs. When jobs fail we get alerts with the stacktrace, which helps a lot during debugging. Early detection results in an early fix, which in turn means minimal impact on users.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4b946036c9f4" width="1" height="1" alt=""><hr><p><a href="https://medium.com/clarisights/monitoring-and-reporting-errors-from-sidekiq-jobs-4b946036c9f4">Monitoring and Reporting Errors from Sidekiq Jobs</a> was originally published in <a href="https://medium.com/clarisights">Clarisights</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Ruby Hotspots: Finding avenues for Memoization and Caching]]></title>
            <link>https://medium.com/clarisights/ruby-hotspots-finding-avenues-for-memoization-and-caching-492c6666807a?source=rss----284fe0b03aa7---4</link>
            <guid isPermaLink="false">https://medium.com/p/492c6666807a</guid>
            <category><![CDATA[performance]]></category>
            <category><![CDATA[ruby]]></category>
            <category><![CDATA[engineering]]></category>
            <category><![CDATA[rails]]></category>
            <category><![CDATA[caching]]></category>
            <dc:creator><![CDATA[Ashu Pachauri]]></dc:creator>
            <pubDate>Mon, 11 Jun 2018 14:07:02 GMT</pubDate>
            <atom:updated>2018-06-11T14:10:52.679Z</atom:updated>
            <content:encoded><![CDATA[<p>Like most fast growing startups building the next generation technical platform, we, at <a href="http://www.clarisights.com/"><em>Clarisights</em></a> engineering, are always trying to balance the speed with which we push out new features and finding ways for the performance of the system to keep pace with this speed.</p><p>There are standard established techniques for scaling any distributed system like <em>horizontal scaling</em>, <em>query optimization</em>, <em>caching </em>etc. A <em>shared distributed</em> <em>cache</em> forms an integral part of any scalable distributed system and can provide huge benefits in both latency and throughput by acting as a very fast middleman between the application and the database.</p><p>However, one of the challenges for caching is to identify the <em>hotspots</em> in code where the benefits for caching would be maximized. Also, sometimes, even the fastest shared cache is not enough for getting desired performance primarily due to fact that the code path being cached is so hot that even the fastest cache becomes a bottleneck. That’s where memoization comes in, where you memoize the information in the application itself and trade off some more memory for performance.</p><p>This article outlines some of the experiements we did for identifying the hotspots in our code and how we mixed memoization with distributed caching to achieve the desired performance.</p><h3>Identifying Hotspots</h3><h4>1. Action level metrics</h4><p>We collect latency metrics for logical units in code like controller actions and use them to identify the primary code paths that we need to target for optimization. We use <a href="http://guides.rubyonrails.org/active_support_instrumentation.html#creating-custom-events"><em>ActiveSupport’s custom events</em></a> for performing the action level instrumentation.</p><p>Although, this approach tells us that there might be inefficiencies in specific code paths, it does not give us enough granularity to identify specific parts of code that can be cached or memoized for better performance.</p><h4><strong>2. Stack dumps</strong></h4><p>There are no alternatives to stack dumps while doing spot analysis for inefficiencies in code. This method runs hand-in-hand with the above method where we have already found a larger logical unit of code which is slow and want to find out the avenues of optimization along that code path. Ruby provides a handy way of trapping interrupts and doing specific actions upon trapping the interrupts. We pass the interrupt using a specific <em>kill</em> signal to the process and performing a stack dump on trapping the kill signal. It’s not too diffiicult to do it ourselves, but <a href="https://github.com/frsyuki/sigdump"><em>Sigdump</em></a> is a nifty little gem that abstracts it out for us. After doing the recommended setup it’s pretty easy to dump a bunch of stack traces and find the top hotspots by counting code occurrences across those stacktraces:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/046575d15d80c2fed77e5a9b5538691c/href">https://medium.com/media/046575d15d80c2fed77e5a9b5538691c/href</a></iframe><h4>3. Test Coverage Numbers</h4><p>Well written tests and comprehensive test coverage form the foundation of any reliable system. But, well written tests can give other important insights too that can be used to performance optimizations. We use <a href="https://github.com/colszowka/simplecov"><em>SimpleCov</em></a> for generating test coverage numbers for our <em>Rails</em> app. A sample SimpleCov report looks like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GDgohMBCHF1ZU-H0PAGaCg.png" /><figcaption>Extract from SimpleCov report</figcaption></figure><p>Looking at the files sorted by <em>Avg. Hits/Line</em> and diving into that source file for the coverage numbers reveals the hottest code paths in that file. An example is:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cG7jLF14NWqENKJbO5GrJQ.png" /><figcaption>SimpleCov coverage extract for a single file</figcaption></figure><p>Now, we can target these hotspots for further optimizations like reducing complexity, caching and/or memoization.</p><p>An option for gathering coverage numbers in production is <a href="https://github.com/danmayer/coverband">Coverband</a>. It also gives the same output as SimpleCov but in production instead of tests.</p><h3>To Cache or to Memoize</h3><p>There is no golden rule for <em>memoization vs caching</em> choice but caching is generally the preferred choice, especially in a distributed system as the same cache can be used across multiple machines and over a larger time duration.</p><p>One of our prominent use cases is running periodic <em>rake</em> tasks in a <a href="https://kubernetes.io/"><em>Kubernetes cluster</em>.</a> The same set of data can be read by multiple rake tasks running in parallel or by the same rake task over multiple runs. That’s why we decided to go with the shared cache on a <a href="https://redis.io/">Redis</a> cluster.</p><h4>Caching is not enough</h4><p>After caching the computations and queries being done in hotspots, we decided to profile our app and rake tasks again, and much to our surprise the reads from redis cache still showed up as top hotspots in multiple occassions.</p><p>That’s when we realized that we would have to go with a combination of caching and memoization, sacrificing some more memory for performance. This enables us to eliminate hotspots in cases where same computation or query is being done in the same process more than once, while the shared redis cache still enables the newer runs to perform those queries faster for the first time.</p><h3>The benefits</h3><p>By using this hybrid caching approach, we were able to eliminate hotspots at some crucial points in our code and were able to <strong><em>reduce having to do network calls to database and to shared cache down by upto 99%</em></strong> in worst case. We were able to optimize the real time path of our Rails app and <strong><em>cut down the worst case query times by more than 50%</em></strong>. Our rake tasks are the ones that benefit mostly from this hybrid caching approach where we were able to <strong><em>reduce the runtimes of some of the worst offenders by upto 60%.</em></strong></p><h3>Lessons learned</h3><p>Finding hotspots in code requires mixing multiple approaches but it’s worth the effort and can provide huge benefits. Caching is a great resource for improving performance characteristics but there are often cases where it’s just not sufficient and needs to be supplemented by memoization.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=492c6666807a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/clarisights/ruby-hotspots-finding-avenues-for-memoization-and-caching-492c6666807a">Ruby Hotspots: Finding avenues for Memoization and Caching</a> was originally published in <a href="https://medium.com/clarisights">Clarisights</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Inflection Point for BI Platforms]]></title>
            <link>https://medium.com/clarisights/the-inflection-point-for-bi-platforms-e30f4fb40f03?source=rss----284fe0b03aa7---4</link>
            <guid isPermaLink="false">https://medium.com/p/e30f4fb40f03</guid>
            <category><![CDATA[digital]]></category>
            <category><![CDATA[advertising]]></category>
            <category><![CDATA[reporting]]></category>
            <category><![CDATA[business-intelligence]]></category>
            <category><![CDATA[marketing]]></category>
            <dc:creator><![CDATA[Clarisights]]></dc:creator>
            <pubDate>Mon, 11 Jun 2018 14:00:45 GMT</pubDate>
            <atom:updated>2018-06-11T14:01:18.063Z</atom:updated>
            <content:encoded><![CDATA[<p>The recent story from <a href="https://www.wsj.com/articles/stop-using-excel-finance-chiefs-tell-staffs-1511346601">WSJ</a> on Finance Chiefs asking their teams to move away from Excel generated a lot of buzz with folks debating the article with a lot of passion on Twitter and other channels, so much so that WSJ had to publish a follow-up to the article presenting the dissenting voice!</p><p>I personally found the debate to be somewhat perplexing since most industries have already moved away from Excel which is a horizontal solution to deeper vertical solutions. Most sales teams used to use spreadsheets as a rudimentary CRM before the CRM Industry took off and solutions like Salesforce came along. CRM choices are now debated with vertical solutions, not Excel! Same with ATS or Applicant Tracking System. Can you imagine going back from a dedicated solution like Lever or Greenhouse to Excel?</p><p>Just as Excel is being disrupted by vertical solutions, another industry which will start seeing disruption is the BI Industry. BI is an all-encompassing term to centralize all the KPI’s a co. needs to track but not only is it being used as an Executive level dashboarding solution, but also by vertical heads to streamline more granular data at a customized vertical level like HR, Finance, Marketing, Sales etc and to customize it for each business vertical requires a small army of analysts + engineers to build and deploy.</p><p>To see how the BI Industry might evolve let’s look at the CRM Industry and how the industry has changed over the years.</p><p>The CRM industry essentially started with Siebel. Early naysayers called the CRM a “glorified database” which while technically accurate completely missed the value of context and workflows.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*8bW2MgUia_mhecKo.png" /></figure><p>The next inflection point was the launch of Salesforce which was the first cloud-based solution and this was seminal in so many ways. Not only was it a big shift in the CRM Industry, it also created SaaS as a viable business concept and shifted the way the Industry looked at selling licenses and established the Internet as a viable distribution channel.</p><p>While we have seen multiple CRM solutions at various ends of the Market the next big shift happened with verticalization in the CRM space. An example for this is Veeva which took the Salesforce platform and customized it for the healthcare industry, taking context to the next level of granularity and applying it not just at a business function level but at an industry level.</p><p>We are at a similar point of inflection for BI Platforms</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*HsgW8hlAzNmE6bOv.png" /></figure><p>With the launch of DOMO, we started seeing the move to cloud solutions from desktop licenses which is why I think we are close to the next stage of inflection which will be vertical intelligence solutions for different business functions.</p><p>One industry I am bullish on is Marketing. Most of the innovation in the Marketing space has been around new ways of targeting and customer acquisition and not enough on data consumption. Marketers continue to use Spreadsheets and generic BI systems for data consumption which have been sold to CIO’s and deployed by Analysts. The channels and sources have grown exponentially from just desktop traffic and search to social, push, mobile etc. The tools available to manage this data have not fundamentally changed at all.</p><p>When I started working in this space in 2005, it was hard to get folks with knowledge of digital marketing so we would hire people from traditional Marketing backgrounds and teach them digital. Any knowledge of SQL was considered a nice-to-have but it was very clear that we were hiring folks based on the depth of their Marketing skills, now we are hiring what I call the “full stack marketer” who not only knows SQL but can also write and deploy code, work with product teams and some knowledge of Marketing is a plus!</p><p>You know how acute the problem is when you see folks at PPC forums asking questions like these.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*YmuCL1T-UGzUgH4Y.png" /></figure><p>The value of vertical solutions can be seen by the success of Google Data Studio — unfortunately, it suffers the same limitations of other platforms the minute you need non-Google data sources.</p><p>This is where the opportunity exists for a well built vertical solution which can tie all Marketing data together with none of the baggage of a generic solution. A vertical solution will be less wide and a lot deeper. A vertical solution will also be sold differently, unlike current BI solutions which are sold to CIO’s, vertical solutions will sell to business heads. With Marketing heads budget overtaking CIO’s this is almost the perfect storm of pressing product need + budgets to pull this off.</p><p><em>This post was originally written on </em><a href="https://www.linkedin.com/pulse/inflection-point-bi-platforms-arun-srinivasan/"><em>LinkedIn</em></a><em> by </em><a href="https://www.linkedin.com/in/arun-srinivasan/"><em>Arun Srinivasan</em></a><em>, Founder and CEO of </em><a href="http://clarisights.com/"><em>Clarisights</em></a><em>. Clarisights helps </em><a href="https://granularhq.com/marketers.html"><em>performance marketers</em></a><em> and </em><a href="https://granularhq.com/agencies.html"><em>digital agencies</em></a><em> eliminate the ETL process and automate data collection, normalization and reporting for all advertising and analytics channels. It also empowers you to get deep insights from marketing data that can create long-term business impact.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e30f4fb40f03" width="1" height="1" alt=""><hr><p><a href="https://medium.com/clarisights/the-inflection-point-for-bi-platforms-e30f4fb40f03">The Inflection Point for BI Platforms</a> was originally published in <a href="https://medium.com/clarisights">Clarisights</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>