<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Reza Esfandani on Medium]]></title>
        <description><![CDATA[Stories by Reza Esfandani on Medium]]></description>
        <link>https://medium.com/@esfandani?source=rss-54b15156b6d6------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*Oj-mNQnmph7yxKTAWqkrBg.jpeg</url>
            <title>Stories by Reza Esfandani on Medium</title>
            <link>https://medium.com/@esfandani?source=rss-54b15156b6d6------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 04:59:16 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@esfandani/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[The story of battling with 10x Traffic]]></title>
            <link>https://medium.com/walmartglobaltech/the-story-of-battling-with-10x-traffic-711589c8878b?source=rss-54b15156b6d6------2</link>
            <guid isPermaLink="false">https://medium.com/p/711589c8878b</guid>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[spark-streaming]]></category>
            <category><![CDATA[a-b-testing]]></category>
            <category><![CDATA[engineering]]></category>
            <category><![CDATA[scalability]]></category>
            <dc:creator><![CDATA[Reza Esfandani]]></dc:creator>
            <pubDate>Fri, 23 Apr 2021 18:07:59 GMT</pubDate>
            <atom:updated>2021-04-23T18:07:59.522Z</atom:updated>
            <content:encoded><![CDATA[<h4>a real-time story</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3eXA49Ou-JEkPz7kIhsJMw.jpeg" /><figcaption>Photo credit: <a href="https://pixabay.com/illustrations/jesus-christ-god-holy-spirit-4779546/">jeffjacobs1990</a></figcaption></figure><p>Last year in my <a href="https://medium.com/walmartlabs/real-time-data-processing-for-monitoring-and-reporting-a-practical-use-case-of-spark-structured-8e4f91f6f3a7">previous post</a>, I described the overall architecture of our real-time spark job pipeline for Walmart’s A/B testing platform, <em>Expo</em>. Since then there have been a lot of changes in our systems. Given the pandemic situation, the number of online shoppers has grown significantly. We observed a 5 to 10 times increase in the amount of traffic ingested by our monitoring pipeline overnight. Thus, I am writing this blog post to briefly go over how we moved towards the cloud smoothly, and the changes we made to our architecture in response to the challenges we faced.</p><h3>Background</h3><p>In nutshell we have a spark structured streaming job ingesting data from Kafka, keeping the state of sessions in memory and checkpoints, and pushing metrics into our time-series database for further evaluation by the experimenters. While having more traffic, overall, is very good for Walmart.com, it means more stress on our monitoring system causing delays when pushing the metrics to the pipeline.</p><p>In this post, I will go through the overall challenges that we had and the actions that we took to handle this enormous traffic. The current system is running for the last six months with almost <strong>zero downtime</strong>. Currently, it processes <strong>150k-250k messages per second</strong> and pushes around <strong>3 million records per minute </strong>out and the delay between the time we start an experiment till we have metrics in our database is less than <strong>3 minutes</strong>. On a normal day, our system runs around <strong>100 –150 concurrent experiments</strong> which amplify the magnitude of data the monitoring system processes.</p><p>To recap I copied the overall architecture here, for more detail you can read <a href="https://medium.com/walmartlabs/real-time-data-processing-for-monitoring-and-reporting-a-practical-use-case-of-spark-structured-8e4f91f6f3a7">my previous post</a>. I also put the new architecture below as a reference for the rest of this post.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*nGIbFPpl_1QKxZCTyQEbhw.png" /><figcaption>The old architecture of the monitoring system</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WuxMCXTbzVEAkcm7i5FzHg.png" /><figcaption>The new architecture of the monitoring system currently running in production</figcaption></figure><h3>The issues and the actions</h3><p>In this section, I categorize the problems that we faced and the actions that we took to overcome the problem. One major change that we made was moving the entire pipeline to google cloud to make sure we will have enough resources to scale out. The previous job was on our private data center and we were running out of our team’s reserved capacity so we decided to move the entire pipeline to the GCP. My focus for the rest of the article will be on the problems that we observed in the pipeline per se.</p><h4><strong>Kafka and dispatcher rate</strong></h4><p><strong>Issue</strong></p><p>In general, each streaming job is either CPU intensive or I/O intensive. Though the nature of our monitor system is different. We have three stages in our streaming pipeline. In the first phase, the job fetches around 150GB(2–3GB/second) of data from Kafka which is a heavy I/O operation. In the second phase, it starts processing the data from the previous stage and create/update the sessions in the system. This step is CPU intensive. The third phase is sending out the metrics to the database. This step doesn’t take too much time.</p><ol><li><strong>Ingestion Rate: </strong>We have a stream dispatcher which evaluates the event and decides to redirect the event to other topics for further processing. This redirection doesn’t take any time and the dispatcher can send data at a higher speed than the consumer. We noticed that dispatcher is sending data to the topic at a higher rate than what we can consume(I/O) and process(CPU).</li></ol><p><strong>Actions</strong></p><ol><li><strong>Drop unnecessary data as early as possible: </strong>One of the early actions that we took was to reevaluate thoroughly the types of beacons that go to the topic. Expo has been running since 2015 and we have a decent number of historical metadata. These data show the types of beacons our experimenters are more interested in. We evaluated the historical metadata against the beacons in the topic and found out around <em>43%</em> of data in the topic never affect the result of the experiments. Thus we filtered out those beacons early in the dispatcher. Filtering in dispatcher is cheaper than the job’s pipeline as every second matters in streaming job and each small milliseconds can accumulate to cause a big delay. This change alleviated the delay and the pipeline was able to catch up.</li><li><strong>Split data by traffic pattern: </strong>We separated the traffic for mobile and web and redirected them to their own topics. This gave us the opportunity to create a separate job for each topic and process each topic independently of the other.</li><li><strong>Add more partitions and computes: </strong>The degree of parallelism in a Kafka consumer group is bounded by the number of partitions. If you have enough resources, more partitions lead to higher throughput. We increased the number of partitions from 360 to 490 for each topic which helped us to fetch data faster from Kafka. We also added the number of computes in our spark cluster to overcome the CPU-intensive part of the job.</li></ol><h4>KairosDB</h4><p><strong>Issues</strong></p><p>The other issue was KairosDB. KairosDB is a wrapper around Cassandra which exposes a rest API to the outside.</p><ol><li><strong>KairosDB ingestion rate: </strong>Before the pandemic, we were happy with KairosDB but with each spike in traffic, we found that we cannot trust it as a reliable component and we need to think seriously about this part of the system. KairosDB Internally uses a local queue to handle the high throughput traffic but during burst traffic, it drops data and also pushes data with delay. We had a few occasions even before the covid-19 that we had a spike either by bots or some traffic burst (like PS5 deals) that KairosDB disappointed us.</li><li><strong>KairosDB memory management: </strong>Another problem with KairosDB was memory exceptions when we queried for a wide range of time. Because of this memory mismanagement, we had to restart the service on multiple occasions.</li><li><strong>Data retention policy: </strong>The metrics in KairosDB are not compressed and it was hard for us to accept requests from other teams to increase the retention period or add more metrics to the current pipeline. We set a two days TTL policy in our real-time pipeline while we had a lot of requests to increase that number. It was even hard for us to add more metrics easily. Every time we had a request to add new metrics, we had to do it with caution. We also had to push back the request like capturing the click and impression metrics for images with our monitoring system as it almost would have doubled our captured metrics.</li></ol><p><strong>Actions</strong>‍</p><ol><li><strong>Evaluate other options: </strong>Even before the traffic increase, we noticed that KairosDB can possibly be an issue. So, we started to replace our time-series database. We evaluated multiple databases and we ended up with two options: <strong><em>Elasticsearch</em></strong> and <strong><em>M3DB</em></strong>. While Elasticsearch is a powerful search engine and its aggregation framework could help us a lot, we found it takes a lot of memory and space compared to other time-series DBs. It was also getting slow to the queries as the size of data grew up. On the other hand, M3DB really shined. We doubled the load to the system by running two separate jobs pushing production data to M3DB and let it run for two weeks. M3DB was able to easily handle the load. The other factor that made M3DB outstanding was the ability to compress the data which helped us to extend the metrics’ TTL to 30 days in our production. We also added metrics like impression and image clicks (with multiple attributes and tags) to the system.</li><li><strong>Give a time for the database to ingest: </strong>We put Kafka as middleware between our spark app and M3DB. Pushing to a Kafka topic is faster than pushing to a database since it just appends to a segment file which can also be done in batch. With this setup, another job can send the metrics to the desired database. For KairosDB we set up a logstash cluster and for M3DB we used an internal java application to ship the metrics to the database. We extensively tested both databases head to head by comparing the results from KairosDB with M3DB through Grafana charts to make sure we don’t see any inconsistency because of this transition.</li></ol><p>With the above changes, we were able to add more metrics without being worried about the stability of the system, increased the retention time to 30 days, and had a zero data drop in metrics for the last six months.</p><h4>Code and config coupling</h4><p><strong>Issue</strong></p><ol><li><strong>convolution of metadata and code: </strong>The previous implementation of UI was directly calling KairosDB to get the metric and calculate the hardcoded formula based on those. It was hard to add a new formula to the system. A formula is a combination of different metrics. For example conversion rate comes from dividing two different metrics, the number of customers divided by the total number of visitors. There are many times that experimenters required a new formula which is a combination of current metrics in our system but in order to add it, the team needed to change the code. This part of the code was convoluted with Expo UI services and it was hard to maintain.</li></ol><p><strong>Action</strong></p><p><strong>Time series database wrapper service(<em>tsdb-ws</em>): </strong>When we decided to switch KairosDB with another unknown data source, we knew that eventually, we need to change the UI code as well. So we decided to unify the interface of communications with our current and future data source (KairosDB, M3DB, Elaticsearch).</p><p>For this purpose, we developed a time-series database wrapper service(<em>tsdb-ws</em>) using koa, a node.js framework. In order to do it, we separated each tenant with a logical keyspace and defined the meaning of each formula for that tenant, and put this relationship into a repository. Each formula can be a template literal. In this case, UI will send the proper payload to make the full expression.</p><p>Besides the formula section, we also have a metadata section in the config file so that provides everything UI needs to render the chart (like the title of the chart, this is a percentage chart or a number chart and etc)</p><p>The UI app hits <em>/&lt;:keyspace&gt; </em>with proper data asking for multiple formulas in batch for a given time period. The<em> tsdb-ws</em> service looks at the list of formulas, <em>finds the dependency graph between formula and metrics,</em> and based on that fetches the entire data with<em> one call</em>. The return values are decorated with metadata of the formula.</p><p>With this architecture, we don’t need any code change. Every time the experimenter needs more formulas to be shown in UI, we can easily add them into the repository, and it will be rendered as a chart in UI.</p><p>Also, this setup helped us to have a smooth transition from KairosDB to M3DB. The UI doesn’t need to know what is the underlying database we are using. When we felt M3DB is ready and we have a proper setup, we just switch the flag in tsdb-ws to use M3DB and UI didn’t even know about it.</p><h3>Wrap up</h3><p>We started to build a highly available monitoring system for Expo around two years ago. For the last two years, our architecture adapted to different circumstances. The current system is working smoothly and we have plans to make sure it’s scalable for our next 10x traffic. We are now confident that this is the right strategy to take for us and for our clients.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=711589c8878b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/walmartglobaltech/the-story-of-battling-with-10x-traffic-711589c8878b">The story of battling with 10x Traffic</a> was originally published in <a href="https://medium.com/walmartglobaltech">Walmart Global Tech Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Real-time data processing for monitoring and reporting — A practical use case of spark structured…]]></title>
            <link>https://medium.com/walmartglobaltech/real-time-data-processing-for-monitoring-and-reporting-a-practical-use-case-of-spark-structured-8e4f91f6f3a7?source=rss-54b15156b6d6------2</link>
            <guid isPermaLink="false">https://medium.com/p/8e4f91f6f3a7</guid>
            <category><![CDATA[a-b-testing]]></category>
            <category><![CDATA[spark-streaming]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[engineering]]></category>
            <category><![CDATA[spark]]></category>
            <dc:creator><![CDATA[Reza Esfandani]]></dc:creator>
            <pubDate>Fri, 03 May 2019 20:20:26 GMT</pubDate>
            <atom:updated>2022-12-06T17:18:26.788Z</atom:updated>
            <content:encoded><![CDATA[<h3>Real-time Data Processing for Monitoring and Reporting</h3><h4>A practical use case of spark structured streaming</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0H1C-JQld4BwZb036hkfJw.jpeg" /><figcaption>Photo credit: <a href="https://pixabay.com/illustrations/particles-color-monitor-wave-1373684/">geralt</a></figcaption></figure><p>In this post, I will explore how we designed and implemented our new real-time monitoring system for Walmart’s in-house AB testing platform, <strong><em>Expo</em></strong>. The system processes billions of records and generates millions of metrics per day. These metrics help experimenters monitor their experiments and how they impact the Walmart site or app.</p><p>In my <a href="https://medium.com/walmartlabs/monitoring-and-alerting-for-a-b-testing-detecting-problems-in-real-time-4fe4f9b459b6">previous post</a>, I gave a small example of how AB testing works. Before pushing each new feature, we expose the new feature to a subset of users. Based on the feedback given from <em>Expo, </em>we either turn on the feature to all users or withdraw the new feature and keep the site as it is. Each experiment has its own objectives. <em>Expo </em>provides a lot of metrics and experimenters can decide which one is more important for them. This can include click-through rate, revenue, or anything that the user wants to be able to measure.</p><h3>Background</h3><p>One of the systems we developed in WalmartLabs is Expo which is our experimentation platform for A/B testing. Expo manages A/B test setup and execution. There are multiple systems that support Expo to accomplish A/B testing and help analysts and experimenters to arrive at a conclusion. One of them is a reporting system that analyzes the data nightly in a batch manner. Another is a monitoring system that runs continuously to give a real-time snapshot of how the experiment is currently working. We will focus on the monitoring system in this article.</p><h4>A dummy scenario to continue the discussion!</h4><p>Now for the rest of this post let’s assume a simplified example as a reference for the rest of the article.</p><p>Let’s say we need to change the color of a blue button to red on the first page of walmart.com and we would like to see how this change affects revenue per visit or conversion rate which is a really important metric in e-commerce. To do so, we create an experiment &lt;exp&gt; with two variations in Expo. In &lt;var1&gt; , which is the control version currently running in production, we have a blue color button. In &lt;var2&gt;the color is red and it’s our desired feature that we would like to eventually push to production. Now when a user visits the home page, if he gets assigned to the experiment, he will either see blue color button (&lt;var1&gt;) or red color button( &lt;var2&gt;) during the time the experiment is running.</p><p>In our example walmart.com is called tenant and has uswm identifier. Expo, and monitoring, support other tenants, such as the Walmart mobile app or Walmart groceries. home_page and item_page are two sensors for visiting those pages that we call measurement point.</p><p>The monitoring system generates many different metrics for each experiment. Metrics are defined by keys, such as uswm.visit.fm.&lt;var1&gt;.item_page.beta. This metric shows the number of unique visits which is qualified for the experiment and visited the item page. The only parts of the naming that might be vague are fm and beta part.</p><p>fm is an abbreviation for factor matched meaning a user only contributes to the value of the metric if he is assigned to &lt;var1&gt; and he is exposed to the treatment. We also call this “qualified” in our system. In our example, the button color change on the home page is the treatment. So a user will increase the example metric if 1) he is assigned to the &lt;var1&gt; 2) he has visited the home page 3) after the home page visit(step 2), at some point he visits the item page.</p><p>If the second step out of the three top conditions is not met, the user will just contribute to uswm.visit.<strong><em>assigned</em></strong>.&lt;var1&gt;.item_page.beta metric.</p><p>It is possible for the fm criteria to be very complicated. For example, the condition could be that the user is visiting the site on a mobile device and chose a specific payment type at checkout. These conditions are specified in the experiment configuration.</p><p>The beta means each sessionId can only contribute to the metric once. We have other metrics without beta showing that if a user goes to the item page multiple times it will increase the metric value multiple times.</p><h3>Overall Architecture</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_45jUOcYmn68PDs9dm4z7A.png" /><figcaption>Image 1: Components involved in the monitoring system</figcaption></figure><p>The above image shows the major components used in our monitoring platform. With this setup we can focus on the bottlenecks and scale up each part horizontally and independent of each other. In the below subsections I will describe how each part works.</p><h4>Kafka</h4><p>When a user hits the page or takes different actions inside a page, logs are generated. These logs are either server logs or beacons being emitted by javascript). Each log has a sessionId that represents a user. Not all the records are useful for generating metrics so we forward the useful logs to their own topic. The job connects to the topic and picks the records one by one and extracts the metrics out of them.</p><h4>Monitoring Job</h4><p>Each instance of the job is a Spark Structured Streaming app that triggers at each minute, reads the data from Kafka, and sinks them in KairosDB.</p><p>To guard against failure, we run 2 mirror instances of the job in separate data centers and let them write the result to our KairosDB independently. Each job tag their metrics by data center. When a failure happens, the system is able to switch to the other data center metrics automatically by changing the tag’s name and UI will serve the correct data.</p><h4>KairosDB</h4><p>KairosDB is the Sink for our streaming job. It is a time-series database backed by Cassandra. It not only keeps our monitoring metrics but also keeps track of the health of the monitoring system. Along with Expo metrics, the monitoring system also populates internal health metrics, such as the size of memory consumed, number of records processed and so many others. We keep dashboards on these metrics in Grafana to be able to debug issues more easily.</p><p>KairosDB is capable of aggregating data based on different tags. Each metric is tagged with different information like the type of device and login state. We were able to generate all metrics and send all combinations to KairosDB but we decided to take the tagging strategy. It allowed us to distribute the complexity between different parts of the system instead of integrating all of them in one location that makes it harder to scale computationally. UI services can query data as granular/coarse as they want by combining different tags.</p><h3>Overall Implementation</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/660/1*NzAW7mKd7eziyPQm1hiQNA.png" /><figcaption>Image 2: High-level view of what job does in each batch</figcaption></figure><p>As mentioned above, the monitoring job is a Spark structured streaming job written in scala running on a minute micro-batch. It includes two stateful phases: one for sessionization and the other consumes the sessionization output for metric aggregation. Structured streaming made the stateful operation part easier to implement by having some higher-order methods like flatMapGroupsWithState. The developer needs to provide a function as an argument passed to flatMapGroupsWithState that gets triggered each time we have data for a key which for example in sessionization is sessionId. The function makes an abstraction and provides some useful parameters for the developer to update the current state. For each key, it provides all data associated to the key for that time range and also gives the developer access to the previous state. The only thing a developer needs to do is focus on the core problem of how to keep the state, what metrics he needs to emit for each phase and how to correct the metrics if the event is out of order, and how to merge the previous state with the current state in the system. The key for the sessionization is a sessionId string and for the metric aggregation phase is (metricName, minute bucket) tuple.</p><h4>Sessionization</h4><p>In the sessionization part, we aggregate the events for each session in a data structure named Session. You can consider each event as an action the user took. We don’t need to keep the entire data and we just need a summary of information in order to generate the metrics. The key for this phase is sessionId. A session is a case class in scala that keeps track of information that we need to generate the new metric and correct the misfired metrics.</p><ol><li>sessionId: The id for the session</li><li>session property: numerical property of the session like last time the session has activity and etc.</li><li>markers: A data structure that keeps track of the events which are important; for example keeps the time the user was exposed to the variation for the first time.</li><li>counts: A data structure that keeps track of sessions’ actions; for example it’s aware of at which time and for how many times the user visited the home page.</li></ol><p>We generate all intermediate metrics with only these four data structures. Some metrics can be directly calculated from markers like the number of assigned users to a variation while others need to be processed by making a Cartesian product of data in markers and counts data structures.</p><p>One example of an output metric of this phase can be:</p><pre>(uswm.visit.fm.123#desktop#in#dc1.home_page.beta.1525356180000,1)</pre><p>As you can see the tags and minute bucket timestamp are serialized in the metric name. This is the output of one session and the value for this metric name needs to be aggregated across all sessions. Making tags as granular as possible gives us this opportunity to reduce the space and time complexity in our real-time monitoring and pass the responsibility of calculating values for different tag combinations to KairosDB.</p><p>Since we populate the metrics for each minute bucket, we need to have access to this information for each metric emitted. That’s why we append the batch time to each metric.</p><h4>Handling out-of-order events</h4><p>Another thing that needs to be considered in real-time system is late data or out-of-order data. For example, let’s say event1 and event2 happened at batch time 10 and 14 respectively but event2 arrives earlier into the monitoring job because of a failure in one of the Kafka brokers. If we don’t consider this out-of-order event, the factor match for this session is considered happened at time 14 while the right value is 10 and this gives us a wrong result. To correct the wrong metrics we need to calculate what metrics have been emitted between times 10 and 14 and correct them by emitting the negative value for those metrics. This resolves the problem but makes computation and space very inefficient if we do it naively and keep track of every state and out of order event. The problem can be solved by using the watermark concept introduced in spark structured streaming. You might be interested to read about it <a href="https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#handling-late-data-and-watermarking">here</a> in detail. In general, it’s a mechanism that tracks the current event time and attempts to clean up the old state, and ignores the very late events.</p><h4>Metric Aggregation phase</h4><p>The input in this phase is all the metrics coming from sessionization phase. Remember in sessionization we only keep track of metrics that belong to each session. In this phase, we aggregate them across all sessions. The key for the stateful operation in this phase is (metric name, minute bucket) tuple. Based on this key, the system just adds the sum of each metric across all sessions.</p><p>The final metrics going to Kairos will be something like this:</p><pre>{ metricName: ‘uswm.visit.fm.123.home_page.beta’, <br>tags=[‘dc1’,’in’,’desktop’], <br>timestamp=1525356180000, <br>count=10000,ttl=’30 days’}</pre><h4>Other metrics</h4><p>Spark has an interface that allows developers to have access to information about the status of the streaming queries. We used the interface StreamingQueryListener to populate information about the monitoring job in KairosDB. These metrics basically are for monitoring the monitoring system!</p><h3>Lessons Learned</h3><p>There are a lot of experiments and challenges that we learned during the development and I would like to share some of the important ones:</p><ul><li><em>Test and Validate</em>: For every feature that we added to the system we extensively wrote multiple units and integration tests and completed it with validation against HDFS to make sure that everything works fine. The only thing that we didn’t want to get undermined was the credibility of the job. Since the system was complicated and adding a new feature might introduce a bug, the tests really helped us save the integrity of the system.<br>We also ran the job for a few days on a non-production cluster after developing (each) feature(s) to see how the new features impact the performance of the system. This lets us test the boundaries and weaknesses of the application and let us focus on real problems.</li><li><em>Embrace functional programming</em>: We love functional programming here in WalmartLabs and for this specific project we intentionally embraced it. Here are some outcomes:<br>- It made our development much faster<br>- It reduced the complexity of the system for us and made the code easier to understand<br>- Immutability of the objects allowed us to debug the code easier<br>- It reduced null pointer exception to zero. We haven’t seen any NPE after we deployed the monitoring job in production.</li><li><em>prototyping is the key for uncertainties</em>: If you doubt about functional and even non-functional aspects of your system the best way to prove the concept is abstracting the use case and making a small prototype of it. It’s really important to realize the fact that spending more time on prototyping and design at the very beginning of the project gives you a better chance of a happy ending project than a final implementation that doesn’t satisfy the requirements and service level agreements.</li><li><em>A chain is only as strong as its weakest link</em>: When you design a multi-tenant system ensure the scalability of each major component in your design and that it can be easily scaled up horizontally during peak traffic. We thought about our design multiple times and flushed it out overtime to make sure it can sustain the peak traffic of Walmart.</li></ul><p>The current monitoring job supports 8 different tenants of Walmart and we are expecting to onboard more tenants. That means every single request of these tenants, after being filtered, goes through this system. The current normal input rate to the application is<strong> 120k</strong> to 2<strong>50k</strong> entries per second and it generates <strong>1 </strong>to<strong> 3 million</strong> metrics per minute.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8e4f91f6f3a7" width="1" height="1" alt=""><hr><p><a href="https://medium.com/walmartglobaltech/real-time-data-processing-for-monitoring-and-reporting-a-practical-use-case-of-spark-structured-8e4f91f6f3a7">Real-time data processing for monitoring and reporting — A practical use case of spark structured…</a> was originally published in <a href="https://medium.com/walmartglobaltech">Walmart Global Tech Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Monitoring and Alerting for A/B Testing — Detecting Problems in Real Time]]></title>
            <link>https://medium.com/walmartglobaltech/monitoring-and-alerting-for-a-b-testing-detecting-problems-in-real-time-4fe4f9b459b6?source=rss-54b15156b6d6------2</link>
            <guid isPermaLink="false">https://medium.com/p/4fe4f9b459b6</guid>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[monitoring]]></category>
            <category><![CDATA[algorithms]]></category>
            <category><![CDATA[a-b-testing]]></category>
            <category><![CDATA[engineering]]></category>
            <dc:creator><![CDATA[Reza Esfandani]]></dc:creator>
            <pubDate>Mon, 04 Jun 2018 18:43:20 GMT</pubDate>
            <atom:updated>2018-06-04T18:43:20.124Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*Q0CKcTfbH_ucDpHSQyoD2Q.jpeg" /><figcaption>Photo credit: <a href="https://pixabay.com/en/red-warning-alert-bulb-light-2708362/">a_roesler</a></figcaption></figure><p>At WalmartLabs we are building Expo, an A/B testing framework for our sites and apps. One of the critical components we have is monitoring and alerting to monitor different aspects of the A/B testing environment. The system needs to detect when there’s a problem with the assignment or with the behavior of users in an experiment and alert the appropriate people.</p><h3>Overview</h3><p>In this article, I will elaborate two of these alerting applications which were built by the A/B testing team for watching our real-time reporting system and to make sure the system and the experiments made in the A/B testing framework works properly as expected.</p><p>Before going further, I would like first to describe the big picture of how A/B testing works so that the audience has a better understanding of the flow.</p><p>In A/B testing each user will be assigned to different variations of an experiment. An experiment can be a UI change, backend algorithm change such as the search algorithm or anything that might impact the user behavior in a way that can be measured. One simple example for UI A/B testing would be having two different designs for buy button. One design has a big red button and the other one is small blue button. Each of these designs is a variation. The variation which is currently running on production is called “control”. After setting up the variations, Expo will split the traffic between these two(or more) variations and based on user behavior, generate metrics for each variation. For example, one metric could be conversion rate, which in our case is the percentage of visitors to our site that make a purchase. If the difference between metric values for those variations are statistically significant, we choose the winner and use it in production.</p><p>For generating these metrics, we have two jobs. One is a real time job that gives us quick, raw data of metrics in real time. The other job is running daily and gives us a more refined calculation from more accurate data sources.</p><h3>Anomaly detection for the number of Visitors</h3><p>One of the metrics that we track is the number of unique users who visit the site that are assigned to an experiment per minute. The real-time job keeps track of this number and pushes the metric to a time-series database. We noticed that there can be a problem in assignment or the real-time job that causes this metric to spike or drop and we wanted to be notified when this happens. One naïve solution for finding the discrepancy was taking the average for a short period of time(<em>average_s_time</em>) like a few minutes back and comparing it with the average for a longer period of time(<em>average_l_time</em>) like the last 3 hours, then we would expect that <em>average_s_time</em> shouldn’t differ from <em>average_l_time</em> by too much. Considering these values are following normal distribution we are able to evaluate <em>average_s_time</em> with 3-sigma test against the values of longer period and decide whether we have an alert for the past few minutes or not. This might seem like a good idea at first but this approach will end up with a lot of false positive alerts that makes the alert system less credible.</p><p>The reason for this behavior is because the data is periodic and the average will fluctuate based on the size of window and the time that we calculate the average value. Therefore, we needed to somehow normalize the data and remove the periodic factor from the data.</p><p>One useful algorithm which has been out for a while is seasonal decomposition of time series by Loess or STL(<a href="https://www.otexts.org/fpp/6/5)">https://www.otexts.org/fpp/6/5)</a> . This algorithm decomposes time series data into three different data sets: seasonal, trend and remainder. The following image shows a sample time series data and how these three datasets would be.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Gtck_2_GoZj0rqWwMJZQew.png" /><figcaption>International airline passengers: monthly totals in thousands for 1949–1960 (<a href="https://tinyurl.com/yb45ca2n">https://tinyurl.com/yb45ca2n</a>)</figcaption></figure><p>As you can see, remainder is the original data without seasonality and trend and you can track the sudden changes in this graph more easily. By applying the previous algorithm on remainder data, we are now able to catch problems in real-time while minimizing false positive alerts.</p><h3>Real Time Experiment Metric Alerting (RTEMA)</h3><p>When an experimenter designs an experiment and divides traffic between them, it’s very critical to know if something is wrong with experiment as soon as possible. For example, there might be a bug in one of the variations that prevents customers from going to the checkout page. We want to ensure that we catch this type of error and the system sends an alert if one of the variations’ data is off by a large amount.</p><p>As mentioned in the overview, Expo tracks different metrics in real-time. Most of them are ratio based meaning it comes from division of two numbers. For example, the conversion rate metric comes from the number of customers who bought something divided by the number of all visitors. However, some metrics, such as assigned visitor count are not ratios. The following algorithm runs every 15 minutes. Each variation has its own metric value generated by the real-time reporting system by minute, hour and day.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3f49d9e6a5db1ed9d63146449563017d/href">https://medium.com/media/3f49d9e6a5db1ed9d63146449563017d/href</a></iframe><p>The above algorithm is easy enough to get so I won’t elaborate it more. The other metric this system watches is the number of visitors for both variations. If it’s not close to the proportion of traffic configured for each of the variations it will send an alert.</p><p>In order for the algorithm to become more practical and reduce the number of false positive alerts, we tweaked the original algorithm with these enhancements:</p><ul><li>As you can see in line 9 each metric has its own threshold instead of using a global threshold. However we had to somehow determine the correct value for it. To do so, we selected some healthy experiments for sampling and and applied the same algorithm from line 8 to find a suitable value for <em>Threshold(M)</em>.</li><li>The other tweak that we made was ignoring low traffic experiments. If the number of visitors are not bigger than a particular number we don’t go further.</li><li>The initial implementation of the algorithm was looking back for the 15 minutes and sending an alert if it was finding any anomalies. We changed the algorithm to run in two rounds. In the first round it finds the possible anomaly metrics for each experiment in the last 15 minutes, then it looks back again for a longer period of time with a smaller threshold(by default it’s <em>Threshold(M)/2</em>) and sends alert if the metrics we get from the first round pass the second round as well.</li></ul><p>With these changes, we reduced false positive alerts dramatically while the system saved us on several occasions by sending true alerts even before the experimenter could catch it.</p><h3>Conclusion</h3><p>Alerting is a critical part of WalmartLabs ecosystem since even small problems can result in a significant loss in customers or revenue. While we have different tools for general performance and errors monitoring, some systems need extra consideration to find unusual behavior. In this article we presented two examples of such needs that we encountered in Expo and how we implemented monitoring and alerting to detect problems in visitor assignment or variation metrics to be able to maintain the quality of the system.</p><p>As the last part of this article of I would like to mention three important points that we followed during development of these algorithms.</p><ul><li>Keep it simple: this is the principal that we take when the approach is heuristic and we would like to develop an algorithm that works with real data. For both of the above algorithms we started with tweaking the data and prototyped a simple version of the algorithm and after finding their weaknesses, we iterated to improve it. This makes the process of development much easier while it’s more practical to see the output of the algorithm against real data and easier to make the right decision for the next step.</li><li>Make it credible: When you develop a custom alerting system, you should take care of false positive alerts. It’s unavoidable to have some certain of false positive alerts because of the nature of real data however we need to consider this factor otherwise people will start questioning the credibility of the entire alerting system.</li><li>Externalize the algorithm’s parameters: This part is probably kind of obvious that we need to extract out the parameters out of the code and give it during the runtime. The reason that I put here is because it’s kind of related to previous section. We need to deal with true positive alerts in real-time here. We may want to tune parameters in real time, or be able to shut down alerts from overwhelming us and then address the root cause of the problem.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4fe4f9b459b6" width="1" height="1" alt=""><hr><p><a href="https://medium.com/walmartglobaltech/monitoring-and-alerting-for-a-b-testing-detecting-problems-in-real-time-4fe4f9b459b6">Monitoring and Alerting for A/B Testing — Detecting Problems in Real Time</a> was originally published in <a href="https://medium.com/walmartglobaltech">Walmart Global Tech Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>