<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by swetha kasireddy on Medium]]></title>
        <description><![CDATA[Stories by swetha kasireddy on Medium]]></description>
        <link>https://medium.com/@swethakasireddy?source=rss-66dd141da64b------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*LXG-YC5I1MF6Pz0MSbzhpw.jpeg</url>
            <title>Stories by swetha kasireddy on Medium</title>
            <link>https://medium.com/@swethakasireddy?source=rss-66dd141da64b------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 15:13:48 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@swethakasireddy/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[How we enabled product and pricing-availability feeds as APIs for external partners]]></title>
            <link>https://medium.com/walmartglobaltech/how-we-enabled-product-and-pricing-availability-feeds-as-apis-for-external-partners-a24448f976ca?source=rss-66dd141da64b------2</link>
            <guid isPermaLink="false">https://medium.com/p/a24448f976ca</guid>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[api]]></category>
            <category><![CDATA[bigquery]]></category>
            <category><![CDATA[gcp]]></category>
            <category><![CDATA[engineering]]></category>
            <dc:creator><![CDATA[swetha kasireddy]]></dc:creator>
            <pubDate>Wed, 02 Sep 2020 22:41:16 GMT</pubDate>
            <atom:updated>2020-09-02T22:41:16.270Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*U9w8lnYrdZdc7rVkrlmiUg.png" /><figcaption>Photo credit: <a href="https://pixabay.com/illustrations/ecommerce-online-sales-sales-1706105/">ReaxionLab</a></figcaption></figure><p>Walmart being the country’ largest grocer, wants customers to be able to shop at their local store digitally and plan their trip even before they choose curbside pickup or delivery.</p><p>We are from Affiliates and OpenAPI engineering team. Our team works on enabling third party integrations to drive traffic and conversion for Walmart. One of our use cases needed a smart and efficient infrastructure to handle variability of products across thousands of stores thereby helping to integrate with many 3rd parties in the future who can help Walmart customers better integrate their lives with Walmart as a retailer.</p><p>In this post, I will give an overview of how we enabled product and pricing-availability information as downloadable feeds for external partners. With these APIs, a Walmart prospective customer can do product discovery for their local store on a 3rd party publisher’s app/website and can directly add their chosen items to Walmart cart. Walmart handles the fulfillment, be it curbside pickup, or delivery.</p><h3>Overview</h3><p>As part of the feeds APIs, we provide a daily snapshot of product metadata, product pricing-availability information across all the active stores. In addition to snapshots, we also provide deltas every 15 minutes. The typical usage pattern for the partners is to use snapshots to bootstrap the product metadata and product pricing-availability information and then incrementally apply deltas as they arrive every 15 minutes.</p><p>Feeds based integration works well for bulk data exchange. For point lookups on individual products we provide real time pricing-availability API that can be used to fetch the most up to date data for an individual item.</p><p>All the APIs are filterable by category paths, the list of which is exposed through our taxonomy API.</p><h3>Key design considerations</h3><p>Following are the key design considerations:</p><ul><li><em>To address bandwidth concerns seen in some of the existing APIs which provide downloading feeds from an on-premises cluster.</em></li><li><em>To ensure security of the APIs we expose to external partners.</em></li><li><em>To ensure that the data we expose is of proper quality as any data discrepancy can cause improper state resulting in add to cart errors for customers.</em></li><li><em>To make the feeds APIs multi tenant. Multi-tenancy is needed to ensure that only the inventory available to a given partner is included in the respective feed or API. Think of it as a category or item level access control for inventory information.</em></li><li><em>Monitoring and alerting of the pipelines to ensure that pipelines are running without any issues.</em></li><li><em>To make the feeds queryable by category.</em></li><li><em>To make the on-boarding process easier for partners.</em></li></ul><h3>Snapshot feed API architecture</h3><p>Snapshot feeds provide a snapshot of product catalog metadata and product pricing-availability information for a given day. Following is the high level architecture for the same:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gJ6-Jyz1F3iSL3s8gMDfCQ.png" /></figure><p>The product catalog and pricing-availability data is collected from a number of upstream systems where the inventory information is managed. Several AirFlow Spark jobs collect the upstream data and transform into consistent, normalized representation in Hive tables.</p><p>Previous versions of this system(with partial functionality) resided on-premises, in internally hosted Hadoop deployment. Interestingly, we’ve observed significant performance improvement as systems leveraged GCP and optimized Spark Jobs— we’ve observed a reduction of 75% processing time as a result of the pipelines running in GCP and optimized Spark Jobs.</p><p>Google cloud storage is used as the storage layer for the feeds to be exposed to external partners. Data is stored in hierarchical structure and partitioned by categoryId — the identifier of the category where a given product belongs. Category based partitioning allows us to utilize GCP storage access control to implement per-partner inventory filtering — only locations of categories that are enabled for given partner are exposed to the partner.</p><p>When partners make an API call for the feeds, a downloadable google cloud storage signed url for the feed data is generated based on the partner configuration and returned in the response. Partners can download the feed data from google cloud storage signed url. This has addressed the bandwidth concerns we have seen in downloading feeds from our on-premises cluster. Security of the feeds APIs is taken care of by enabling necessary authentication and authorization mechanisms at API Gateway and also by providing a google cloud storage signed url for data feeds which expires after a pre-configured time.</p><p>Following is a reference document of google cloud storage signed url.</p><p><a href="https://cloud.google.com/storage/docs/access-control/signed-urls">Signed URLs | Cloud Storage | Google Cloud</a></p><p>Monitoring and alerting pipeline that runs in AirFlow sends slack/email alerts if there are any failures in any of the DAGs.</p><h4>As part of the data quality check for the snapshots, we do the following:</h4><ul><li><em>Any anomalies in the data generated are alerted if the percentage difference exceeds a pre-configured threshold.</em></li><li><em>We also ensure that the feed file has the correct format, schema and if all the required values are present or not as part of data quality check. We are looking to build additional functionality on top of this data quality check pipeline as we observe trends over the course of time.</em></li></ul><h3>Delta feed API architecture</h3><p>Delta feeds APIs provide deltas in product pricing-availability information. The downloadable feed file has hourly dumps and the dumps are updated every 15 minutes. The high level architecture looks like the following:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hDMJMlDPR6T9NI12uWyM3g.png" /></figure><p>The delta feeds pipeline runs entirely in GCP. We listen to a set of upstream Kafka topics to determine product pricing-availability. Spark structured streaming jobs running in dataproc cluster are used to listen to kafka topics and store the transformed data in our base tables in BigQuery. AirFlow DAG that has deltas pipeline runs every 15 minutes and deltas are derived by running sql queries on our base tables and snapshot table in BigQuery and stored back to our deltas table in BigQuery.</p><p>Deltas are queried from BigQuery and updated to per-partner downloadable feeds every 15 minutes. Deltas are also stored in hierarchical structure based on the categoryId and are queryable per-categoryId.</p><p>When partners query for deltas, a downloadable google cloud storage url is provided similar to what was described in snapshot feed architecture. Partners can also get the deltas at daily/hourly frequency from a particular date-hour.</p><p>Monitoring and alerting ensures that the streaming jobs and the deltas pipeline in AirFlow is running continuously and deltas are generated without any delays. Slack/email alerts are provided for any failures/delays. Automatic Streaming job restarts when the jobs go down is taken care of by the scheduler that monitors the streaming jobs.</p><h4>As part of the data quality check for the deltas, we do the following:</h4><ul><li><em>Data quality check that runs every 15 minutes compares the deltas we generate against a realtime price-availability API and alerts any discrepancies seen. This helps us to proactively troubleshoot any data discrepancies seen and apply fixes if needed.</em></li><li><em>We generate a snapshot of the deltas table from BigQuery once a day and compare it with an upstream table and alert any discrepancies seen. The purpose of this is to troubleshoot the discrepancies and apply back the mismatches to BigQuery deltas table if needed.</em></li><li><em>We also ensure that the deltas feed file has the correct format, schema and if all the required values are present or not as part of data quality check. We are looking to build additional functionality on top of this data quality check pipeline as we observe trends over the course of time.</em></li></ul><h3>Conclusion</h3><ul><li>A very important part of the feed pipelines is to make sure that the feed data provided is as accurate as possible. Having a proper design in place by talking to various upstream teams helped us in achieving this.</li><li>A reusable framework for monitoring and alerting of the pipelines helped us in ensuring that the pipelines run without any issues.</li><li>Data quality check layer helped us to identify any discrepancies seen in the data we generate as part of the feeds and take necessary actions.</li><li>Leveraging GCP for running our data pipelines and optimized Spark Jobs also helped speed up our final snapshot generation with the whole process completing under 2 hours.</li><li>Leveraging GCP for providing downloadable feeds(with google cloud storage signed URL) helped us in addressing some of the bandwidth concerns seen while downloading feeds from on-premises infrastructure.</li><li>Important consideration has been given to ensure security of the APIs exposed to external partners by enabling necessary authentication and authorization mechanisms for the APIs at API gateway and google cloud storage signed URL for data feeds which expires after a pre-configured time.</li><li>On boarding new partners is made easier by going with an API based approach for providing feeds.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a24448f976ca" width="1" height="1" alt=""><hr><p><a href="https://medium.com/walmartglobaltech/how-we-enabled-product-and-pricing-availability-feeds-as-apis-for-external-partners-a24448f976ca">How we enabled product and pricing-availability feeds as APIs for external partners</a> was originally published in <a href="https://medium.com/walmartglobaltech">Walmart Global Tech Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[CTR Optimization via Thompson Sampling]]></title>
            <link>https://medium.com/walmartglobaltech/ctr-optimization-via-thompson-sampling-83df19fa577f?source=rss-66dd141da64b------2</link>
            <guid isPermaLink="false">https://medium.com/p/83df19fa577f</guid>
            <category><![CDATA[engineering]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[mab]]></category>
            <dc:creator><![CDATA[swetha kasireddy]]></dc:creator>
            <pubDate>Tue, 14 Jan 2020 22:58:07 GMT</pubDate>
            <atom:updated>2020-01-14T22:58:07.959Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/853/1*yBaZYikBdxRsUe0-8ZEfcA.jpeg" /><figcaption>Photo credit: <a href="https://pixabay.com/photos/slot-machine-one-armed-bandit-play-425713/">khfalk</a></figcaption></figure><p>Walmart Labs is a data-driven company. Many business and product decisions are based on the insights derived from data analysis. I worked in Expo which is the A/B Testing platform for Walmart. As part of the platform, we developed a feature to optimize content based experiments based on CTR on Walmart.com. CTR(click-through rate) is the percentage of impressions that resulted in a click. The objective of CTR optimization is to dynamically allocate traffic to variations that have a higher CTR, while allocating less traffic to variations that have lower CTR. This feature has been developed based on the multi-armed bandit (MAB) content testing paradigm.</p><h3>Multi Armed Bandits</h3><p>By analogy, a “multi-armed bandit” is a machine that has multiple arms to pull each with a different expected value of payoff. The player needs to figure out which arm gives the highest payout. The phase of pulling out different arms in order to identify the one with highest expected payoff is termed “<strong>exploration</strong>”. The phase of pulling the arm that’ most profitable so far in hopes of maximizing the total return is termed “<strong>exploitation</strong>”.</p><p>The goal of “multi-armed bandit” problem is to construct a strategy to strike a balance between exploration and exploitation with the goal of maximizing a particular criteria. The multi-armed bandit has both exploration and exploitation phases, which alternate in an adaptive manner based on the arms’ ongoing performance. This paradigm fits well with website optimization where we would want to dynamically shift the traffic to winning variation based on a particular criteria like CTR.</p><p>The decision to explore or exploit is dependent on the bandit algorithm being used and in our case the bandit algorithm being used is Thompson Sampling.</p><p><em>Some advantages of MAB over A/B testing:</em></p><p>For short term campaigns/applications the cost for waiting for the A/B test results make MAB a better choice so that large gains can be made by exploiting as early as possible.</p><p>MAB can also be used to automate testing process and avoid repeated intervention by analysts in order to perform repeated A-B tests.</p><p>MAB can be used to capture dynamic changes when the item being tested changes significantly over a period of time invalidating the results of an A/B test. In this case, MAB provides an optimal strategy by continuously exploring.</p><h3>Goal of the Algorithm</h3><p>The goal of the bandit algorithm is to dynamically allocate traffic to variations that are performing well, while allocating less traffic to variations that are underperforming. To begin with, it should display all variations to a random selection of users, and measure which variants are clicked on more frequently. Over time, it will use these observations to infer which variation has the higher CTR. Then, once the estimation of the CTR becomes more precise, it will gradually move traffic towards the winning variation.</p><h3>Optimization Workflow</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*w-IPTUPHf1Vti9Q5mzEUmA.png" /></figure><p>We provide a UI for the Optimization experiment setup. The 3 key configurations that we provide as part of the Optimization experiment setup are the following:<br> 1. Optimization Wait period(This is to allow the optimization metric to “warm up”: to get a reasonable number of hits, so we don’t reallocate traffic based on too little data. Default is 24 hours.)<br> 2. Optimization Interval(This is a configuration to setup the frequency of the optimization. For example for home page, we can optimize often say every 1 hour vs for a category page where views are low the optimization engine can optimize every few hours. The default is 1 hour.)<br> 3. Minimum traffic per variation(This is to prevent optimization engine from making a “too hasty” traffic adjustment in favor of one variation and to allow to explore other variations as well. Default is 10%.)</p><p>Initial traffic allocation is at equal percentage for all the variations of the experiment. Once the experiment is started, the assignments of the visitors would happen for each variation depending on the allocated percentage. A realtime spark structured streaming job is used to collect clicks/impressions per minute for each variation and stored in a Time Series Database. The clicks/impressions are provided every hour to the bandit model to generate the new traffic percentages. Any traffic adjustment itself is done after Optimization Wait period is met and at the frequency of optimization Interval.</p><h3>Explore/Exploit using Thompson Sampling</h3><p>The basic idea of Thompson Sampling in a Multi Armed Bandit scenario is that the algorithm starts with a prior belief about the expected reward. During a trial, a random sample is drawn from the prior of each arm. The arm chosen is the one that has the highest sample which provides the best arm. Post feedback, the prior belief is updated to a posterior belief. Since we use conjugate priors, the prior and posterior take the same form and we can continue the sampling process treating the posterior as the new prior. The algorithm ensures that the arm with larger expected reward is exploited more while intermittently exploring the under performing arms.</p><p><em>The following section describes the explore/exploit approach we took and how we used Thompson Sampling to do dynamic traffic adjustments of the Optimization experiments.</em></p><p>We keep track of positive and negative observations. Here positive observations are clicks. Negative observations are (impressions-clicks).<br>The beta distribution is used as prior. We use conjugate prior which means that the posterior distribution is the same family as the prior.<br>Original priors are chosen as beta(20,980). Based on the 2% CTR seen historically we have used 20 positive observations and 980 negative over 1000 impressions as the prior. Before 1000 impressions we don’t want to make any movement either way. As the amount of data grows the effect of priors is negligible.</p><p>Each time the algorithm receives a feedback, we update the positive and negative observations. Whenever we need to access the posterior distribution, we compute the parameters for the beta distribution. We will use these parameters to draw a random sample from the beta distribution.</p><p>We also apply a decay factor. Everything up to 7 days ago gets weight 1, the week before gets weight 𝛌, the week before that gets weight 𝛌², and so on. Currently, the 𝛌 is set as 0.8.</p><p>Around 200K samples are from posteriors of beta binomial distribution and we construct an empirical distribution by counting the proportion of times that a given variation was chosen. The traffic split is based on the percentage of winning for each variation. For example: If 30% of time variation 1 wins as against 70% of time for variation 2. The traffic split is 30% for variation 1 and 70% for variation 2. In the steady state we expect the winning variation to have 100% of the traffic.</p><p>Following is a sample code on how Thompson Sampling can be used to do dynamic traffic adjustments based on observed clicks and impressions.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c24291a19f798eeec50c0fb260a4426a/href">https://medium.com/media/c24291a19f798eeec50c0fb260a4426a/href</a></iframe><h3>Conclusion</h3><p>We have done a few pilot runs and observed a favorable lift in the CTR. We also observed that short term campaigns where large gains can be made by exploiting as early as possible by reducing the exploration phase is a good candidate for MAB.</p><p>As a next step we are looking to enhance the system to support optimization by other metrics like add to cart rate, conversion rate, revenue, minimizing bounce rate. We are also looking to enhance the feature beyond Content based treatment type. If there are multiple components in a page to continuously optimize, we are looking to use MAB approach as a framework to automate the optimization process.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=83df19fa577f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/walmartglobaltech/ctr-optimization-via-thompson-sampling-83df19fa577f">CTR Optimization via Thompson Sampling</a> was originally published in <a href="https://medium.com/walmartglobaltech">Walmart Global Tech Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How we built a data pipeline with Lambda Architecture using Spark/Spark Streaming]]></title>
            <link>https://medium.com/walmartglobaltech/how-we-built-a-data-pipeline-with-lambda-architecture-using-spark-spark-streaming-9d3b4b4555d3?source=rss-66dd141da64b------2</link>
            <guid isPermaLink="false">https://medium.com/p/9d3b4b4555d3</guid>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[lambda-architecture]]></category>
            <category><![CDATA[spark-streaming]]></category>
            <category><![CDATA[a-b-testing]]></category>
            <category><![CDATA[engineering]]></category>
            <dc:creator><![CDATA[swetha kasireddy]]></dc:creator>
            <pubDate>Wed, 27 Jun 2018 00:08:30 GMT</pubDate>
            <atom:updated>2018-06-27T00:08:30.477Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Rur3q_Gp8NCZvp-EK0-QOg.jpeg" /><figcaption>Photo credit: <a href="https://pixabay.com/en/analysis-analytics-business-chart-1841158/">Pexels</a></figcaption></figure><p>Walmart Labs is a data-driven company. Many business and product decisions are based on the insights derived from data analysis. I work in Expo which is the A/B Testing platform for Walmart. As part of the platform we built a data ingestion and reporting pipeline which is used by the experimentation team to identify how the experiments are trending. In this blog I would like to give a little primer on how we built the data ingestion and reporting pipeline with Lamda Architecture using Spark which provides code reusability between the Streaming and the batch layers, key configurations for the deployment and a few troubleshooting tips.</p><h3>Use Case</h3><p>The use case is to take the click stream events, aggregate them based on the session id and generate metrics such as unique visitors, visits, orders, revenues, units, bounce rates, site error rates, performance metrics etc for both assigned and qualified experiments. Following is our architecture that accomplishes that:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*D5TEZ5tmp9MQoErOA3K2aw.png" /></figure><h3>Lamda Architecture</h3><p>We have been running a Lambda architecture with Spark for more than 2 years in production now. The Lambda architecture provides a robust system that is fault-tolerant against hardware failures and human mistakes. The resulting system is linearly scalable by scaling out instead of scaling up. We needed a lambda architecture as there is no guarantee that the data will arrive exactly once and there can be noise in the data and to protect against data that arrives late due to network or server instability. Lambda architecture handles these issues by processing the data twice, once in the realtime streaming to give a quick view of the data/metrics that get generated and second time in a scheduled batch process to give a more reliable view of the data/metrics. One key consideration that we had with our lambda architecture was to make our writes to KairosDB idempotent in order to be able to reprocess the data and guarantee exactly once processing semantics.</p><h3>Why Spark/Spark Streaming</h3><p>Spark/Spark streaming improves developer productivity as it provides a unified api for streaming, batch and interactive analytics. Overall, Spark has been a flexible, robust and scalable data processing engine for our use case. Spark Streaming provides efficient fault-tolerant stateful stream processing. It provides high throughput and good integration with data stores like Cassandra, Kafka, Hadoop etc.</p><h3>Stats</h3><p>Here are some statistics about our data pipeline:</p><ul><li>~70k events per second on an average</li><li>~250k events per second during peak hours</li></ul><h3>Details</h3><p>We use Spark Kafka Direct for our streaming that guarantees exactly once processing semantics with checkpointing enabled for recovery. Batch processing is built on top of core spark.The above mentioned metrics are calculated and stored in KairosDB, a time series DB. Also, the user session information is stored in HDFS as Hive External tables from the batch processing. The user state is maintained for 12 hours in streaming and in batch the user state is maintained as long as the user is active.</p><p>RDDs in the streaming and the batch processing are processed as follows:</p><p><em>Parsing and Validation</em>: This is done in a series of map(), flatMap(), filter(), groupByKey(), reduceByKey() and leftOuterJoin() RDD transformations which gets reused across Streaming and Batch jobs.</p><p><em>Data Enrichment and state maintenance</em>: Data enrichment and state maintenance is done under updateStateByKey() in Streaming and mapPartitions() in case of batch processing with the business logic for merging the sessions and enriching the session with additional information getting reused across batch and streaming jobs. In streaming the state is maintained in memory and in the checkpoint as part of the functionality provided by updateStateByKey(). In batch processing, the state is maintained in Hive External Tables in HDFS. We also calculate rolling hour and 24 hour unique visitors using reduceByKeyAndWindow() with inverse function in the Streaming job.</p><p><em>Metric Generation and Save to KairosDB</em>: Metrics are generated using flatMap() and reduceByKey() after the Sessionization and data enrichment and are saved to KairosDB using a kairosDB client. The write to KairosDB happens in forEachPartition() action. The logic for the save gets reused in both the Streaming and batch processing.</p><p>In the stream processing, the incoming beacons for each micro batch are grouped by sessionId and sorted by timestamp. Session state is updated by processing the sessions from the current micro batch and merging them with the session state from the previous micro batch. Metrics generated after sessionization are reduced and then stored to KairosDB. We store the offsets in Zookeeper in case we need to restart the job from the last successfully processed offsets. For accuracy, our transactions update the offsets in the same transaction when updating the results. The following code sample gives an overview of the same.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2c1c364aead1aca898b296a41e0dbcc5/href">https://medium.com/media/2c1c364aead1aca898b296a41e0dbcc5/href</a></iframe><p>The following code sample has the reusable functions that are called from the Stream processing and batch processing for filtering and populating the valid beacons, populating grouped and sorted sessions, processing Stateful Sessions , generating the metric values, reducing the metrics and storing the generated metrics to KairosDB.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b4230cd2932daad18885f6ccd54a6c46/href">https://medium.com/media/b4230cd2932daad18885f6ccd54a6c46/href</a></iframe><p>The following is a code sample of the batch processing that does the Stateful Sessionization, Metric Generation and Save to KairosDB and the above functions that are called from Stream Processing are called from batch processing as well thus providing code reusability between the batch and stream processing.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8b82021cc553805c9f6495d03e996cab/href">https://medium.com/media/8b82021cc553805c9f6495d03e996cab/href</a></iframe><h3>Streaming Configuration and Deployment</h3><p>We use a standalone cluster for Spark Streaming job deployment. For HA/DRwe run the Streaming Job in 2 different regions. The Streaming job from only one region writes to Kairos cluster at any point of time and the job in the second region keeps running by maintaining the state with no writes to kairos. If there is any issue with the job in the first region or if we need to do any maintenance to the cluster or upgrade the code or if there are any issues with the Kafka topic in one region, we switch the writes to Kairos from the job running in the second region. This gives us higher availability of the Streaming Job with session state being maintained and the data written to kairos is continuous.</p><p>Our streaming job runs with a micro batch of 60 seconds. The correct micro batch size is chosen depending on the processing times. We started with a batch window size of 10 seconds and observed processing times and increased it incrementally until the processing times are completed within the micro batch time period and there is no scheduling delay or it’s only increasing occasionally and recovers quickly. The ideal number of executors/cores is dependent on the application by considering various factors like number of peak events per second, maximum allowed lag and the buffering capabilities of the streaming source which can be arrived at by testing in a pre-production environment. The right amount of parallelism is between 2–3 times the number of cores and needs to be arrived at iteratively by testing the job with various configurations. Another key consideration is to set spark.memory.fraction and spark.memory.storageFraction to the right values. Based on various tests, it is observed that spark.locality.wait=0s is good for the job performance. We use kryo serialization in our Spark Jobs for better performance. When upgrading the application code, the application needs to be shutdown gracefully with no further records to process.</p><p><strong><em>Job Monitoring and Automatic Job Restarts for Streaming</em></strong></p><p>We use a python script that runs every 5 minutes to monitor the streaming job to see if its up and running. We monitor if there is any lag, total delay, number of failed jobs vs number of successful jobs and take appropriate steps if it exceeds a certain threshold. We query for these metrics from Spark UI end point http://streamingUIURL:4040/metrics/json and http://streamingUIURL:4040/api/v1/applications/&lt;appId&gt;/jobs. If the job is down for some reason or the number of failure jobs are greater than the number of successful jobs for the past 5 minutes or if any of the task is stuck due to any infrastructure issues then the job gets restarted automatically and we receive an alert email.</p><h3>Troubleshooting the streaming job</h3><p>We use Spark streaming UI to troubleshoot any issues with the job. Following is a screenshot of the Streaming UI. Ideally the scheduling delay should be 0 or even if it increases, it should recover quickly within acceptable time periods. The processing time should be less than or equal to the microbatch time period. The Kafka Direct stream consumption parallelism is equal to the number of Kafka partitions and the number of partitions need to be adjusted if there is not enough parallelism for the reads.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*t_74B6EvvCF5fNsKym-7ow.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7uX2KN_7-ie0kxAnuim2Jg.jpeg" /></figure><p>In case of increased processing times, individual batches can be drilled down to see which stage is taking time as shown below. Shuffle reads/writes should be as less as possible and there should not be any data skew. If the shuffles are taking a lot of time, then one way of fixing that is by increasing the cores and the parallelism.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tYNOgavG7YPlsAG6_NAAeg.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Tj10psSC2Q1K881knRc5eA.jpeg" /></figure><p>DAG visualization as shown below is used to determine if there are any pipelining operations that are separated by shuffles. Performance is good if all the operations go through a single pipeline without any shuffles. It also gives an insight if any of the RDDs are cached. The cached RDDs are denoted by a green highlight. Caching this RDD means future computations on this RDD can be accessed from memory thereby improving the performance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LtJz30Afb0H6YE9lobmpyA.jpeg" /></figure><p>The storage section of streaming UI as shown below provides an insight to see if the cached RDDs are regularly cleared or not. Ideally cached RDDs should be cleared at regular intervals to avoid any performance bottlenecks.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*csuMR3SzRm-O_rsA0MZ5kg.jpeg" /></figure><p>The executors tab of the Web UI as shown below also provides information about the number of active tasks, utilization of the cores, task GC time, shuffle reads/writes. If there is a huge difference between active tasks vs cores allocated then there is under utilization of the cluster and the core allocation need to be adjusted accordingly. Thread dump is used to drill down on possible performance issues.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DjTS3iD5OmPKD9iXhrhIkQ.png" /></figure><h3>Batch Configuration and Deployment</h3><p>We use Mesos as our cluster manager for batch jobs and use HDFS for state maintenance and intermediate file storage. The batch processing job runs every 30 minutes and processes any new data available in HDFS. The data is stored in Hive external tables. We configured the executor memory depending on the amount of data that is getting processed in each batch. We also use shuffle operations like reduceByKey, groupByKey, repartition, coalesce, cogroup, join etc as part of the batch job, the configs like spark.executor.memory, spark.cores.max, spark.default.parallelism were adjusted according to the job requirement iteratively.</p><h3>Troubleshooting the batch job</h3><p>A few slow local tasks could cause a huge performance impact when one stage needs to finish before the next one can start. The detailed stage/tasks link in the Spark UI can be used to identify slow running nodes, nodes with resources problems, skew in data partitioning which can be identified by looking at the input size/records, or a small number of tasks taking significantly longer time to execute than the others. Drilling down into slower running tasks we can determine if the slowness is in writing data, reading data, or computation. If the processing is slow, it can be due to not enough resources and we would need to focus on how much memory and cpu has been allocated for the executers and also the total number of cores allocated for the job.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UQAiUxRtjKgjCs9qjGsxvA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*n3SFFilBJTQ3CqorUaR3rg.png" /></figure><h3>Conclusion</h3><p>To recap, here are some points covered in this article</p><ul><li>Building the data pipeline for A/B testing with the lambda architecture using Spark helped us to have quick view of the data/metrics that get generated with a streaming job, and a reliable view from a scheduled batch process.</li><li>Using Spark/Spark Streaming helped us to write the business logic functions once, and then reuse the code in a batch ETL process as well as a streaming process which helped us lower the risk for errors resulting from duplicate code bases and also helped with the developer productivity as it provides a unified api for streaming, batch and interactive analytics.</li><li>We focused on having a stable stream/batch processing application first before focusing on the throughput. The performance of the applications were improved by tuning Spark’s serialization, memory parameters, increasing the number of cores and parallelism iteratively.</li><li>Spark Streaming/Batch UI provides very good information on the performance bottlenecks like shuffles, data skew, slow running tasks due to resource issues, task GC time, shuffle reads/writes, slow running stages, storage and other information to help troubleshoot the jobs.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9d3b4b4555d3" width="1" height="1" alt=""><hr><p><a href="https://medium.com/walmartglobaltech/how-we-built-a-data-pipeline-with-lambda-architecture-using-spark-spark-streaming-9d3b4b4555d3">How we built a data pipeline with Lambda Architecture using Spark/Spark Streaming</a> was originally published in <a href="https://medium.com/walmartglobaltech">Walmart Global Tech Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>