<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Vikas Singh on Medium]]></title>
        <description><![CDATA[Stories by Vikas Singh on Medium]]></description>
        <link>https://medium.com/@vikassingh1000?source=rss-8e404c3e3309------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*yN7N5wu0bTRkMUu304kY9g.jpeg</url>
            <title>Stories by Vikas Singh on Medium</title>
            <link>https://medium.com/@vikassingh1000?source=rss-8e404c3e3309------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 16:16:48 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@vikassingh1000/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Priority Queue in AWS]]></title>
            <link>https://vikassingh1000.medium.com/priority-queue-in-aws-22fa8c4b46b8?source=rss-8e404c3e3309------2</link>
            <guid isPermaLink="false">https://medium.com/p/22fa8c4b46b8</guid>
            <dc:creator><![CDATA[Vikas Singh]]></dc:creator>
            <pubDate>Thu, 13 Apr 2023 21:27:18 GMT</pubDate>
            <atom:updated>2023-04-13T21:27:18.795Z</atom:updated>
            <content:encoded><![CDATA[<p>Queues are a very common and powerful tool to handle asynchronous operations or act as buffer between the flow of data in a system.</p><p>Messages are added to the queue, and they are processed in the order they were received(with some exception).<br>This allows for a decoupling of processes and allows them to communicate with each other.</p><p>AWS has two services for queue<br>1- Amazon Simple Queue Service (SQS): SQS is a fully managed message queue service that enables us to decouple and scale microservices, distributed systems, and serverless applications. <br>2- Amazon MQ: Amazon MQ is a managed message broker service for Apache ActiveMQ and RabbitMQ that makes it easy to set up, operate, and scale message brokers in the cloud.</p><p>Both have their own different configurations/flavors depending on requirement ex SQS has two options Standard and FIFO.</p><p>AWS doesn’t have priority queue. Today this article will explain one approach to implement scalable, high performant and available priority queue in AWS.</p><p>First below is the flow of Autonomous store’s user entry flow</p><figure><img alt="System sequence diagram" src="https://cdn-images-1.medium.com/max/1024/1*cyVQ-RlarmJD6olMHIO4hw.png" /><figcaption><strong>System sequence diagram</strong></figcaption></figure><p>Retail cloud system will receive user entry notification from two different. They don&#39;t have common id which can be used to link them. Retail Cloud is suppose to link them based using Country, storeId, gateNo (Store has multiple gates and these gates have QR code scanner) and order of entry.</p><p>AWS FIFO does provide the orders but it does not provide API to receive message per group. So this service doesn&#39;t fit to here.</p><p>We could use DynamoDB as priority queue. DynamoDB’s supports sparse index. GSIs are sparse by default.</p><p>Below is the AWS component diagram.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/690/1*4h9073396040iSCyn9k6xw.png" /><figcaption>AWS component Diagram</figcaption></figure><p>Below is the table structure.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/664/1*NBcWmRUen7KD5gdLHrmSYg.png" /><figcaption>Relevant columns and Indexes</figcaption></figure><p>Steps flow</p><p>1- User Entry linking handler will create new entry for each new user without shooperID.</p><p>2- When AI system sends request, AI user recognition handler will use GSI and get the oldest record (using ScanIndexForward=False in query API)</p><p>3- If entry not found it will wait for 1 second and again retry</p><p>4- Once Record found, it will add new column shopperID and drop country_storeid_gateNo column( these will be atomic transaction using update_iteam API and UpdateExpression field).</p><p>5- As soon as column country_storeid_gateNo is drop, this row will be removed from GSI and next time this row will not be return at step -4 for subsequence request.</p><p>Thanks for Reading.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=22fa8c4b46b8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[System Communication for Transactional events on AWS]]></title>
            <link>https://vikassingh1000.medium.com/system-communication-for-transactional-events-on-aws-1b7c17a86fdd?source=rss-8e404c3e3309------2</link>
            <guid isPermaLink="false">https://medium.com/p/1b7c17a86fdd</guid>
            <dc:creator><![CDATA[Vikas Singh]]></dc:creator>
            <pubDate>Sun, 14 Aug 2022 19:03:05 GMT</pubDate>
            <atom:updated>2023-01-05T19:07:28.720Z</atom:updated>
            <content:encoded><![CDATA[<p>Building blocks of an Enterprise/Complex system are multiple sub Systems.</p><p>The top two ways for a system to interact with each other are polling and event/push-based. These methodologies can be implemented in a variety of ways, including file sharing, APIs, event/messaging, and so on.</p><p>This blog will cover event-based interactions. If these systems are housed in the same Cloud, the pub-sub approach would be the preferred option. To accomplish it, all cloud providers provide a variety of services. AWS provides SNS-SQS, Kinesis, and Kafka hosted on ECS as options.</p><p>Depending on the system deployment structure, such as multicloud or cloud and on-premises, API calls or Webhooks may be viable options.</p><p>While system-designing recently, I came across a scenario in which one system was deployed on Azure and another system was deployed on AWS.</p><p><strong>Brief system introduction</strong>: We were attempting to implement a frictionless store in order to improve the customer experience ( very much similar to Amazon Go)</p><p>The system’s tailored HLD is shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/802/1*Ua_vpYGfzeMXj9HVwnb6Lw.png" /><figcaption>Trailered version of Enterprise design</figcaption></figure><p><strong>Brief on source system (ML Engine):</strong>The Source System records user shopping behaviours using a variety of data sources, such as video feeds and weight sensors. The final event is generated after ML corelates all of these events.</p><p>These events are received and processed by the Retail Cloud system. These are shopping events that must be handled in the order listed.</p><p>Ex:</p><p>1 — The Customer picked 5 Coca-Colas</p><p>2 — The Customer returned 3 Coca-Colas</p><p>Below is the proposed design to integrate these two systems (this flow does not cover the unhappy scenario).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/971/1*p9c4CasHVFxqM98sha7SKg.png" /></figure><p>The source system will push the data using the AWS HTTP API. These data will be received by the FIFO SQS. It is similar to the payload shown below.</p><pre>{<br>   &quot;eventSequenceID&quot;:&quot;&quot;, # Unqiue number generated by Source system<br>   &quot;storeId&quot;:&quot;&quot;,<br>   &quot;shopperNumber&quot;:&quot;&quot; # Source system assign unique shopperNumber to each customer<br>   .....<br>   .....<br>}</pre><p>eventSequenceID is mapped to the de-duplication ID and the storeId mapped to the SQS groupId (these are two values used by SQS FIFO to enforce <a href="https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/FIFO-queues-exactly-once-processing.html">exactly-once processing</a>). These events can’t be processed concurrently because they are related to customer shopping ( i.e. transactional).</p><p>One can argue that the groupId field should be customerId or shopperNumber. This approach will be more parallel than the previous one.🤨</p><p>The consumer system recognises customers based on the email address they registered with the app, and two people can use the same app to enter the store at the same time (the app generates QR codes that require to be scanned at the entry gate). Using the camera, however, the source system generates a unique shopperNumber for each customer.</p><p>As a result, a source system may have two distinct customers, but both may be the same registered user on a consumer system.</p><p>It’s a story for another blog about how these customers’ IDs are linked/stitched in two different systems. 😅.</p><p><strong>Event Flow:</strong></p><ol><li>ML Engine generates event and sent it to Retail cloud HTTP API.</li><li>HTTP API pass this event to FIFO SQS and response success event.</li><li>SQS push this data to lambda based on FIFO logic.</li><li>Lambda parse data, add message attributes and push it to FIFO SNS</li></ol><p>Below is the SAM template used for this.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b72c4118f0d54d4d108c0be2fdbbcdc4/href">https://medium.com/media/b72c4118f0d54d4d108c0be2fdbbcdc4/href</a></iframe><p><strong>Scalability of System:</strong></p><ul><li><strong>API Gateway</strong>:<strong> </strong>AWS HTTP Gateway is a serverless, highly scalable service. It’s not concerning pointabout it.</li><li><strong>FIFO SQS: </strong>According to AWS documentation, SQS FIFO can support up to 3,000 messages per second, per API function (SendMessageBatch, ReceiveMessage, or DeleteMessageBatch). Which is more than enough for the system’s needs. How? These are extremely sophisticated stores and requires good investment, so on higher side consider 25 % of stores modernize in each country, there will be 500 stores per country. (The blog will go into detail about why counting is done per country rather than globally.)<br>The maximum number of concurrent customers in these modest retail establishments would be ten (if they all got lucky at the same time :D).<br>Each customer generates events (product pickup or putback) every five seconds on average.</li></ul><blockquote>Putting all number together (500 *10) / 5 =&gt; 1000 event per second per country.</blockquote><ul><li><strong>Lambda</strong>: This is a scalable service; the top limit of the AWS account should be properly set. It can be increased from the initial 500 if 80% of it is spent.</li></ul><p><strong>Limitation:</strong> Currently, HTTP API gateway and SQS integration do not support passing path parameters. Below is organization’s standard URL format.</p><p><strong>ex: HTTPS://{host}/{platform}/{service}/{version}/{country}/{resource}</strong><br>Country value will not be transferred to SQS and, as a result, to additional micro services.<br>To address this, each route was assigned a unique SQS.<br>A naming convention can be used to identify the country, and the country name can be appended to the end of the SQS name.<br>It is possible to accomplish this in a clean manner by creating SQS for each country and then creating a lookup DB or Store parameters to store the SQS to country mappings.<br>This appears to be cleaner, but it adds milliseconds to the lookup time in Lambda. To save time, a convention-based approach can be chosen over a configuration-based approach.</p><p>Thats it for today. Thanks for reading.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1b7c17a86fdd" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Chunk based S3 file processing.]]></title>
            <link>https://vikassingh1000.medium.com/chunk-based-s3-file-processing-376e6416903c?source=rss-8e404c3e3309------2</link>
            <guid isPermaLink="false">https://medium.com/p/376e6416903c</guid>
            <category><![CDATA[s3]]></category>
            <category><![CDATA[large-file]]></category>
            <category><![CDATA[slicing]]></category>
            <category><![CDATA[aws-lambda]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[Vikas Singh]]></dc:creator>
            <pubDate>Mon, 31 Jan 2022 17:42:49 GMT</pubDate>
            <atom:updated>2022-01-31T17:42:49.905Z</atom:updated>
            <content:encoded><![CDATA[<p>Serverless has been the buzzword for a long time now. The term originated from the idea that the infrastructure used to run your backend code does not need to be provisioned and managed by you and your team.</p><blockquote>In this blog, I will walkthrough, how Stepfunction and Lambda can be used to process the large file in minimal resources and without timing out.</blockquote><p>Lambda is the favorite choice for processing. But it has a limitation of 10 GB memory and a max 15 min run time.</p><p>If we have not designed our system properly, this limitation could be a road blocker. We may choose to run such processes on Glue Python also. Which could solve the timeout issue but not memory issues and it may be bit expensive in some cases.</p><p><strong>Situation</strong>:</p><p>Recently I had given prerequisites, where the source would be sending deltas every 15 min. 95–98% time these files would be <br>in a few Mbs. But a few times, it could be in GBs also(Yes, it’s a crazy source system 😆).</p><p><strong>Solution</strong>:</p><p>We could write processing in Lambda but for large files, it would fail. We may choose to give larger memory to lambda but it’s not good resource utilization in the case of smaller files.</p><p>To handle this, I decided to chunk the files into smaller sizes and process them. Boto3 API document is not very helpful in explaining how we can use get API to read a chunk of a block of S3 file. We could pass offset and bytes to read in range parameters.</p><blockquote>It follows the HTTP specification. <a href="https://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.35">https://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.35</a>.</blockquote><p>Below is the proposed solution flow</p><figure><img alt="proposed_solution" src="https://cdn-images-1.medium.com/max/724/1*2Sl5wg5Q4bNDazhI4H3v0g.png" /><figcaption>Proposed Solution</figcaption></figure><p>This API returns StreamingBody, which can be used to stream the chunks.</p><p>Below is the high-level approach to accomplish it.</p><ol><li>Start Step Function.</li><li>Trigger Lambda with offset as 0 and other parameters.</li></ol><p>2.1. Invoke Boto3 S3 API with parameter range as value 0 and 50*1024*1024 (read from env config). For more detail on range refer example in <a href="https://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.35">https://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.35</a>.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8c4aa1ef706b931b6d36875f39a51991/href">https://medium.com/media/8c4aa1ef706b931b6d36875f39a51991/href</a></iframe><p>2.2. Get the StreamingBody object and read data.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/42b5202fe0ea2a99b9fe1c39582d269f/href">https://medium.com/media/42b5202fe0ea2a99b9fe1c39582d269f/href</a></iframe><p>2.3. Set has_next to False if it has returned fewer bytes.(As nothing to read further)</p><p>2.4. Create an array of bytes by using splitlines API. <br>2.5. Iterate over the lines[:-1]. Skip last line, as it may be partially read.</p><p>2.5.1 Convert them into UTF-8. Execute business logic.<br>2.5.2 Update offset for next invocation.</p><p>2.6. Return offset and has_next</p><p>3. If has_next is True, again trigger lambda with new Offset.</p><p>4. Using Map state of Step Function, execute business logic parallel on all generated files.</p><p>Below is the Step Function flow.</p><figure><img alt="Large file processor" src="https://cdn-images-1.medium.com/max/381/1*4_BeKVXa2gQx30qHcaBsGg.png" /><figcaption>Large File processor</figcaption></figure><p>Let’s jump into Python implementation. To manage the opening and closing of streams, S3ChunkReader class will use Python Context manager.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1acbb0e636c89795fc1c62a7919b7cc3/href">https://medium.com/media/1acbb0e636c89795fc1c62a7919b7cc3/href</a></iframe><p>Lambda will use this class using with the statement. Below is the input JSON for Lambda.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/915a2291584b82e3486403dfb372f953/href">https://medium.com/media/915a2291584b82e3486403dfb372f953/href</a></iframe><p>Below is the lambda. This will use S3ChunkReader and create a new file in S3.<br>It will keep appending the generated file. This will be used in the next step for parallel processing.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1125c253de01245fe3c554fd389b957e/href">https://medium.com/media/1125c253de01245fe3c554fd389b957e/href</a></iframe><p>After completion of the file splitter step function. Below would be the output.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c28c0c92e5b4a34c82abcf6e4392b270/href">https://medium.com/media/c28c0c92e5b4a34c82abcf6e4392b270/href</a></iframe><p>generated_file_list attribute could be used as input for record processor map state. Map State helps to execute a max of 40 instants of Lambda or configured state.</p><p>Below is the Step function definition.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/54c3625e4eb4400a47c5b1d75bb30d57/href">https://medium.com/media/54c3625e4eb4400a47c5b1d75bb30d57/href</a></iframe><blockquote>The illiterate of the 21st century will not be those who cannot read and write, but those who cannot learn, unlearn and relearn — Alvin Toffler</blockquote><p>Please share your thought on this, Happy Learning.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=376e6416903c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to avoid BroadcastNestedJoin in Spark]]></title>
            <link>https://medium.com/analytics-vidhya/how-to-avoid-broadcastnestedjoin-in-spark-372d20b8e82d?source=rss-8e404c3e3309------2</link>
            <guid isPermaLink="false">https://medium.com/p/372d20b8e82d</guid>
            <dc:creator><![CDATA[Vikas Singh]]></dc:creator>
            <pubDate>Wed, 31 Mar 2021 10:08:15 GMT</pubDate>
            <atom:updated>2021-04-01T07:50:58.204Z</atom:updated>
            <content:encoded><![CDATA[<p>Joins are one of the crucial operations in Spark jobs. Spark supports the following types of joins</p><ol><li>Inner Join</li><li>Left outer</li><li>Right outer</li><li>Full outer</li><li>Left Anti</li><li>Left Semi</li></ol><blockquote>In this article, we will explore, performance issue with Broadcast Nested Join and how to avoid it.</blockquote><p>Each one has its use cases. Spark Cost-based Optimizer select the best join strategy based on various parameters, like:</p><ul><li>Type of Join</li><li>Join conditions(equi, between etc.)</li><li>Size of Data</li><li>Hint</li><li>Spark configurations like of spark.sql.autoBroadcastJoinThreshold</li><li>And many more…</li></ul><p>Based on the aforementioned parameters, Spark selects one of the join strategy listed below:</p><ul><li>Broadcast Hash Join</li><li>Shuffle hash join</li><li>Shuffle sort-merge join</li><li>Broadcast nested loop join</li></ul><blockquote>For more information on it, please refer below link.</blockquote><p><a href="https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/SparkStrategies.scala">spark/sql/core/src/main/scala/org/apache/spark/sql/execution/SparkStrategies.scala at master · apache/spark</a></p><p>For most of the scenarios, spark selects the correct join strategy, but there is some scenario, where selected strategy is not best. We will discuss one of them in this article and ways to fix it.</p><p>For demonstration purposes, I will use the below dataset.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ebFn4JoEoqKIVPZ_6hGiew.png" /><figcaption>Employee and Bonus tables</figcaption></figure><p><strong>Purpose</strong>: Calculate bonus for employees.</p><p><strong>How: </strong>Join employee and bonus table based on <strong>min_salary≤salary ≤ max_salary</strong></p><p><strong>Expected Outcome</strong>: Calculate bonus in optimal time.</p><p>For better performance, as bonus table is small it should be eligible for Broadcast Hash Join.</p><p>First we will load the data into DataFrames and then join them using between function. Below is the code snippet for it.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/fe3443d3897dc66233e07f64785008ba/href">https://medium.com/media/fe3443d3897dc66233e07f64785008ba/href</a></iframe><p>Below is the prepared physical plan by Spark, In this, it has used ‘BroadcastNestedLoopJoin’ Join strategy. Which is slow. We will understand this with the below example</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/50e21e53d5fb15748718bdc2d54aae72/href">https://medium.com/media/50e21e53d5fb15748718bdc2d54aae72/href</a></iframe><p><strong>Ex</strong>: In a real-time scenario, Let’s consider one partition has 100K lines, and broadcasted table has 20K records.</p><blockquote>Ball part estimate: Optimize the size of the partition in Spark is 100 MB(approx). For 12 columns and 100K records, file size would be 104 MB )</blockquote><p>So with the above calculation, for each partition, BroadcastNextedLoopJoin will iterate over 2,000,000,000 times. Which could lead to performance issues. Here we have considered that all partitions are evenly distributed(Maybe before this step, we have to perform other join or group by or window operations). Which is not the case all the time.</p><p>The same join strategy will be used, if we use a join condition similar like below:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/87a3037f023733ef5db1f6396c9f62d7/href">https://medium.com/media/87a3037f023733ef5db1f6396c9f62d7/href</a></iframe><p>Spark will use Broadcast Hash Join if we add one equal condition in join like below.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/db4c7ef2b8faa79102cb76d1d05fa624/href">https://medium.com/media/db4c7ef2b8faa79102cb76d1d05fa624/href</a></iframe><p>But how we will get this new column, which we can use in the joining condition and it does not impact the joining outcome.</p><p>Let us try to add default columns in both Dataframe and use it in the joining condition.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/467b314f3c99ef1d63d03fd0cc8a9932/href">https://medium.com/media/467b314f3c99ef1d63d03fd0cc8a9932/href</a></iframe><p>From the physical plan, it can be seen that Spark has ignored this new column in the query plan, which may be due to some internal optimization.</p><p>Let us try to construct a constant column with the help of existing columns.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e7414854c1c77b88e5712be10e169038/href">https://medium.com/media/e7414854c1c77b88e5712be10e169038/href</a></iframe><p>We have used the salary column and multiply it with 0 and used the resultant column in joining. Now Spark is using BroadcastHashJoin.</p><p>What if we don’t have a numerical column in Data. We could create a dummy Dataframe with one record and cross join with both Dataframe.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b341a60a93b06babf1d124ac8fd1a879/href">https://medium.com/media/b341a60a93b06babf1d124ac8fd1a879/href</a></iframe><p>In above code ,Spark has considered temp_clm in joining. which converted BroadcastNestedLoopJoin to BroadcastHashJoin. <br>As cross join is being performed with one record, its impact will be negligible.</p><p>We could use any approach depending on our convenience and force the spark to use BroadcastHash join, which would be faster in comparison to BroadcastNestedLoopJoin .</p><p>Please share your thought on this, Happy Learning.</p><blockquote>“Once you stop learning, you start dying”<br> — Albert Einstein</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=372d20b8e82d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/how-to-avoid-broadcastnestedjoin-in-spark-372d20b8e82d">How to avoid BroadcastNestedJoin in Spark</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Multithreading/Parallel Job in Aws Glue]]></title>
            <link>https://medium.com/analytics-vidhya/multithreading-parallel-job-in-aws-glue-a291219123fb?source=rss-8e404c3e3309------2</link>
            <guid isPermaLink="false">https://medium.com/p/a291219123fb</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[spark]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[glue]]></category>
            <category><![CDATA[multithreading]]></category>
            <dc:creator><![CDATA[Vikas Singh]]></dc:creator>
            <pubDate>Mon, 21 Sep 2020 20:09:19 GMT</pubDate>
            <atom:updated>2020-09-23T13:58:22.995Z</atom:updated>
            <content:encoded><![CDATA[<h3>Multithreading/Parallel Jobs in AWS Glue</h3><p>On AWS based Data lake, AWS Glue and EMR are widely used services for the ETL processing. AWS Glue is a specialized service for ETL. It has various components which help us to build a robust ETL system.</p><blockquote>In this article, I would like to explain the multi-threading approach in AWS Glue Job to process data faster.</blockquote><p>This article assumes that you are aware of the basics of the following technologies:</p><p>1 — AWG Glue</p><p>2 — Spark</p><p>3 — Python</p><p>ETL processing will always have data-wrangling jobs, which will read staged data, perform validations, lookups and write it into the target.</p><p>Sometimes it happens that the job reads stage, performs some data enrichment, does validation, and then use this half-cooked data into two different target processing. As depicted below:</p><figure><img alt="Using partial cooked data for two different Target" src="https://cdn-images-1.medium.com/max/608/1*sH9QfSMZMMfk5HlKXjenaA.png" /></figure><p>More often than not this could turn into a performance bottleneck. This could be handled by one of the following approaches:</p><p>1 —Create two jobs - one for each target and perform the partial repetitive task in both jobs. This could run in parallel, however this could be inefficient.</p><p>2 — Split the job into 3, first will perform all the common tasks and stage the data. Then, the other two jobs will execute target specific tasks. This will increase I/O and partially send us into the MR era.</p><p>3 — Use multithreading to submit two jobs in parallel to Spark.</p><p>This article will discuss the 3rd approach that I have used and it works like charm.</p><blockquote>Spark context is thread safe, setLocalProperty can be used to set thread local configurations.</blockquote><p>For demo purpose, I will use below stage and dim tables:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/498/1*OtaMVdeGlGuZOqiZII3YMg.png" /><figcaption>Stage and dim sample data</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/684/1*1l-gkViUHoWr7Asc08KccA.png" /><figcaption>Target tables</figcaption></figure><p>To configure Spark in FAIR mode, we need to update spark.scheduler.mode to FIFO and provide fairscheduler.xml for the pool’s configuration file location in <strong>referenced file paths </strong>.</p><blockquote>fairscheduler.xml is must, otherwise Glue container will skip the scheduler setting</blockquote><p>I have tried without it and got below log in job.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*be__OD6_dG6TMKmCgChFSA.png" /></figure><p>Below is the sample code to configure:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/355a2a8c73d48d9689f0e88c230ff3c7/href">https://medium.com/media/355a2a8c73d48d9689f0e88c230ff3c7/href</a></iframe><p>I have used below fairscheduler.xml. In this, it has created two pools (queues) with different weights.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5cae8aa6ede1306ee6aa74ee9704e757/href">https://medium.com/media/5cae8aa6ede1306ee6aa74ee9704e757/href</a></iframe><p>Glue will create 3 pools, 2 configured one and 1 default. This can be seen in below logs. It will assign both jobs into different pools and thus enabling processing both jobs in parallel.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*T6mFA5qDAVfkg-KU7me2ng.png" /></figure><blockquote>“Being a student is easy. Learning requires actual work.”<br> — William Crawford</blockquote><p>Please share your thought on this, Happy Learning.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a291219123fb" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/multithreading-parallel-job-in-aws-glue-a291219123fb">Multithreading/Parallel Job in Aws Glue</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>