<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Amit Ramachandran on Medium]]></title>
        <description><![CDATA[Stories by Amit Ramachandran on Medium]]></description>
        <link>https://medium.com/@amit3jim6?source=rss-4834b7407b5b------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*1NEj-nPxHsNFDnQyV-0Ezw.jpeg</url>
            <title>Stories by Amit Ramachandran on Medium</title>
            <link>https://medium.com/@amit3jim6?source=rss-4834b7407b5b------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 04:02:05 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@amit3jim6/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Adaptive Cards : Generating Synthetic Training Images]]></title>
            <link>https://medium.com/@amit3jim6/adaptive-cards-generating-synthetic-training-images-2c8d173f5a8f?source=rss-4834b7407b5b------2</link>
            <guid isPermaLink="false">https://medium.com/p/2c8d173f5a8f</guid>
            <category><![CDATA[adaptive-card]]></category>
            <category><![CDATA[opencv]]></category>
            <dc:creator><![CDATA[Amit Ramachandran]]></dc:creator>
            <pubDate>Mon, 20 Sep 2021 03:35:29 GMT</pubDate>
            <atom:updated>2024-05-16T01:29:28.084Z</atom:updated>
            <content:encoded><![CDATA[<h3>Adaptive Cards : Generating Synthetic Training Images</h3><p>According to the Microsoft <a href="https://adaptivecards.io/">official adaptive card website</a>, it has been explained as <em>Adaptive Cards are platform-agnostic snippets of UI, authored in JSON, that apps and services can openly exchange.</em></p><p><a href="https://github.com/Microsoft/AdaptiveCards/tree/main/source/pic2card">Pic2Card</a> is a solution for converting adaptive cards GUI design image into adaptive card payload JSON. Embarking towards tweaking the model that we built we needed more training images, to solve this we decided to generate our own training dataset.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kQH5S4ona3RBBV-QZ7W0nw.png" /><figcaption>comparison of an actual adaptive card sample and generated image</figcaption></figure><p><strong>Mainline</strong> :</p><p>Adaptive cards are actually built similar to UI layouts with containers to hold a wide range of card elements like image, textbox, radiobuttons and submit buttons and many more.</p><p>Similar to that to generate an synthetic adaptive card created an layout using the pillow library in python that will act as an container to hold the card elements in place.</p><p>Some challenges with this approach was to handle</p><ul><li>fixed height and width of layout</li><li>finding the annotations for the card elements</li><li>card elements size that are greater than the layout size</li></ul><p><strong>Solution</strong> :</p><p>To solve the issues mentioned above, rather than accomodating different card elements to a layout of fixed size, it made sense to merge the card elements first and create borders for the card elements which made the layout to be dynamically generated.</p><p>To find the annotations all that was needed was to know the dimensions of the card elements and the padding dimensions between the image elements which gives the spacing between them. Using these numbers, a <em>generate_annotations </em>function was created to create annotations in Pascal VOC format.</p><p>To generate random sample images, used the random module to select specific set of card elements ( can be found in configuration ).</p><p><strong>Additional features :</strong></p><p><strong><em>Background colour for the layout </em></strong>: Opencv lets us add colour to the borders that are created. Can use the config variable CANVAS_COLOR to add more colors or use the ones provided.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/528/1*mbDHwJvIfRxivj2DIdMe5A.png" /><figcaption>Grey background</figcaption></figure><p><strong><em>Zipfile for generated samples and annotations </em></strong>: Generating a huge number of samples will overload the CPU for sometime and creating a zipped folder for the sample images handled that issue gracefully.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/618/1*p73o0ZvJOcbqgldpp2UKtw.png" /></figure><p><strong><em>Card elements arrangement</em> :</strong> There will be requirements where user might need the sample image generated should have all the elements arranged vertically. This can be achieved by setting the config variable ELEMENT_COUNT_THRESHOLD for how many input elements needs to be vertical.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Bct4dK2uvjWeXm_Fq2DUUA.png" /><figcaption>Default Threshold when lesser than the card elements argument given while running the command, layout expands to right.</figcaption></figure><p><strong><em>Command line arguments</em> </strong>: Added command line arguments using argparse.</p><p>python -m commands.generate_synthetic_dataset</p><p><em>To generate zip files:</em><br>python -m commands.generate_synthetic_dataset -z True</p><p><em>To create images with coloured backgrounds</em><br>python -m commands.generate_synthetic_dataset -c &quot;ROSE&quot;</p><p><strong>Improvements needed </strong>: Background colour works well and good for light shades, but for dark shades doesnt work since the contrast of both the text and the background gets overlapped.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/494/1*zVw4yjaba8tbJQYCTyv1PA.png" /><figcaption>Text box element is very difficult to make out and is ambiguos for background navy blue</figcaption></figure><p><strong>Repo Links :</strong></p><p><a href="https://github.com/BigThinkcode/AdaptiveCards">GitHub - BigThinkcode/AdaptiveCards: A new way for developers to exchange card content in a common and consistent way.</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2c8d173f5a8f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Partitioning in Spark]]></title>
            <link>https://medium.com/@amit3jim6/partitioning-in-spark-4bbf75971ec7?source=rss-4834b7407b5b------2</link>
            <guid isPermaLink="false">https://medium.com/p/4bbf75971ec7</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[spark]]></category>
            <dc:creator><![CDATA[Amit Ramachandran]]></dc:creator>
            <pubDate>Mon, 05 Aug 2019 03:18:10 GMT</pubDate>
            <atom:updated>2020-05-13T03:58:10.940Z</atom:updated>
            <content:encoded><![CDATA[<p>In the world of distributed computing, we are bound to write efficient programs to reduce the latency, which can be achieved by following certain best practises. Lets see how data partitioning reduces latency and improves parallelism.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/0*pLnXeTS24PZIvCI5.jpeg" /></figure><p>We can see the difference in the time taken by tasks like Reading 1 MB sequentially from memory whereas reading the same from the disk/SSD and while sending bytes across network.</p><p>Lets see another chart of the latency numbers in humanized version to infer the time difference</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/0*JL-t0dKl0CO_xrVT.jpeg" /></figure><p>Now that we have a better intution about the latency lets relate, What’s the link between latency and partitioning and how they are connected.</p><h3>What is Partitioning?</h3><p>Simply put, the data within an RDD is split into many partitions and sent across the worker nodes.</p><p>Note that the data partition never span across the nodes. If a worker node with a partition is lost then data is retrieved by recomputing that RDD via its Parent RDD’s (thats where the resilient part of the RDD comes from)</p><p>Custom partitioning is only possible when working with Pair RDDs since partitioning is done based on keys</p><h3>Types of Partitioning</h3><p>By default, when a spark job starts the number of partitions is equal to the total number of cores on all executor nodes.</p><p>So for example if all of the machines in your cluster have 4 cores and you have 6 worker nodes then that means the default number of partitions you can start with may be 24.</p><p>Spark reads data from HDFS and number of partitions will be determined by numbers of input split for the file in HDFS. If there are N input splits then there will be N partitions. if we are reading a textfile the the input splits will be set by TextInputFormat.</p><p>My Input file size is about 114.5 GB which is 114500MB divided by 32 (fs.local.blocksize default value is 32 MB)) gives around 3579 partitions. But if the lines in your file is too long (longer than the block size), amount of partitions would be smaller.</p><h4>The three types of partitioning are</h4><ul><li>Hash partitioning</li><li>Range partitioning</li><li>Custom partitioning</li></ul><h4>Hash Partitioning</h4><p>Uses Java’s Object.hashCode method to determine the partition as partition = key.hashCode() % numPartitions.</p><p>We will go with an example of some tuples (data) and convert it to RDD (hashpartrdd) and then partition it.</p><p>We can mention the type of partition we want on the data by .partitionBy() method. The number of partitions we want can be done by passing it inside the HashPartitioner() method.</p><p>Note that partitionBy() is a transformation, so it always returns a new RDD. Therefore it is important to persist otherwise, partitioning is repeatedly applied, which involves shuffling each time the RDD is used.</p><p>Hash partitioning tries to spread around the data as evenly as possible over all of the partitions based on the keys. Here we can see the data entering the partions as below.</p><p><strong>Lets assume this in a distributed system then the data on partition 0, 1, 2, 3 are sent to worker nodes 1, 2, 3, 4.</strong></p><ul><li>The partition (0) consists of the tuple (1,1), (1,2), (1,3) → worker node 1</li><li>The partition (1) consists of the tuple (1,4), (3,1), (2,2), (2,3) → worker node 2</li><li>The partition (2) consists of the tuple (3,1), (4,2), (3,3) → worker node 3</li><li>The partition (3) consists of the tuple (4,1), (6,2), (4,3) → worker node 4</li></ul><p>If we want to group the elements by key across the worker nodes, then we will be calling the groupByKey method.</p><p>Notice that above groupByKey method actually groups the data over the network across the worker nodes (shuffling). This paves way to more time taken to compute the resulting grouped RDD.</p><h4>Range Partitioning</h4><p>To eliminate the above shuffling across the network, we can use Range paritition where the keys of our Pair RDD’s can be segregated within certain ranges. Lets consider the previous example “data”.</p><p>RangePartitioner will sort the records based on the key and then it will divide the records into a number of partitions based on the given value.</p><p>import org . apache . spark . RangePartitioner</p><p>Vector ( ( 1 , 1 ) , ( 1 , 2 ) , ( 1 , 3 ) , ( 1 , 4 ) , ( 3 , 1 ) , ( 2 , 2 ) , ( 2 , 3 ) , ( 3 , 1 ) , ( 4 , 2 ) , ( 3 , 3 ) , ( 4 , 1 ) , ( 6 , 2 ) , ( 4 , 3 ) )</p><ul><li>The partition (0) consists of the tuple (1,1), (1,2), (1,3), (1,4) → worker node 1</li><li>The partition (1) consists of the tuple (3,1), (2,2), (2,3), (3,1), (3,3) → worker node 2</li><li>The partition (2) consists of the tuple (4,1), (4,2), (4,3) → worker node 3</li><li>The partition (3) consists of the tuple (6,2) → worker node 4</li></ul><p>When ran in cluster we can see the difference in time taken comparing GroupByKey on hash and range partitioned RDD’s</p><p><strong><em>We can notice that the range partitioned data is more skewed while the hash partitioned data is more evenly distributed across the nodes, yet the compute time is less for range partitioner because it eliminates the data shuffle between nodes.</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/0*sNJ9MZCcimmCo9z2.png" /></figure><p><em>Spark’s Java and Python APIs benefit from partitioning in the sameway as the Scala API. However, in Python, you cannot pass a HashPartitioner object to partitionBy; instead, you just pass the number of partitions desired (e.g., rdd.partitionBy(100)).</em></p><h4>Custom Partitioning</h4><p>To implement a custom partitioned, we need to extend the Partitioner class and implement the three methods which are</p><ul><li><strong><em>numPartitions</em></strong> : Int : returns the number of partitions</li><li><strong><em>getPartitions(key : Any) : Int</em></strong> : returns the partition ID (0 to numPartitions-1) for the given key</li><li><strong><em>equals()</em></strong> : This is the standard Java equality method. Spark will need to test our partitioner object against other instances of itself when it decides whether two of our RDDs are partitioned the same way!!</li></ul><p>we have defined a class named CustomPartitoner. lets partition our data using this class.</p><p>The custom partitioner has partitioned as below.</p><ul><li>4 % 4 = 0 — — -&gt; The partition (0) consists of the tuple (4,2), (4,1), (4,3)</li><li>1 % 4 = 1 — — -&gt; The partition (1) consists of the tuple (1,1), (1,3), (1,4)</li><li>2 % 4 and 6 % 4 = 2 -&gt; The partition (2) consists of the tuple (2,2), (2,3), (6,2)</li><li>3 % 4 = 3 — — -&gt; The partition (3) consists of the tuple(3,1), (3,1), (3,3)</li></ul><p>Thus we can customize the condition on which the partition of data happens.</p><h4>Invoking Partitions</h4><ul><li>By calling PartitionBy method on the RDD</li><li>Certain transformation functions will partition the data under the hood.</li></ul><p>For example the join function invokes hash partitioning by default while the sortByKeys function invokes range partitioning by default</p><p>Lets see how sortByKey invokes partitioning.</p><p><strong>Functions that can invoke some kind of partition under the hood</strong></p><ul><li>join</li><li>leftOuterJoin</li><li>rightOuterJoin</li><li>groupByKey</li><li>reduceByKey</li><li>cogroup</li><li>foldByKey</li><li>combineByKey</li><li>mapValues (if parent has a partitioner)</li><li>flatMapValues (if parent has a partitioner)</li><li>filter (if parent has a partitioner)</li><li>sort</li><li>partitionBy</li><li>groupWith</li></ul><p>Notice that map and flatMap function isnt listed above, but mapValues and flatMapValues are. This is because the map and flatMap functions have the ability to change the keys of the RDD hence has the power to change the partitioned RDD.</p><p><strong><em>NOTE : Before using a map or Flatmap over a large transformed RDD its good to cache that RDD. Partitioning will not be helpful in all applications for example, if a given RDD is scanned only once, there is no point in partitioning it in advance. It is useful only when a dataset is reused multiple times in key-oriented operations such as joins.</em></strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4bbf75971ec7" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What is Spark?]]></title>
            <link>https://medium.com/@amit3jim6/what-is-spark-a343b075ad7b?source=rss-4834b7407b5b------2</link>
            <guid isPermaLink="false">https://medium.com/p/a343b075ad7b</guid>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[spark]]></category>
            <dc:creator><![CDATA[Amit Ramachandran]]></dc:creator>
            <pubDate>Mon, 15 Jul 2019 18:20:04 GMT</pubDate>
            <atom:updated>2019-07-15T18:20:04.847Z</atom:updated>
            <content:encoded><![CDATA[<p>Spark is a fast and general-purpose cluster computing system for <em>real-time processing</em>. It was developed at the AMPLab at U.C. Berkeley in 2009 and later donated and open-sourced by Apache. It has a thriving open-source community and is the most active Apache project at the moment.</p><p><strong>Speed: In-memory processing</strong></p><p>Unlike Hadoop, Spark uses RAM for processing data and this makes it 100x faster than that of Hadoop.</p><p><strong>Support: Developer-friendly API’s</strong></p><p>Spark provides for lots of instructions that are a higher level of abstraction than what MapReduce provided. Currently spark gives support for Scala, Python, R and Java API to program and as well can be integrated with Yarn, Mesos, Kafka, Cassandra, Hbase, MongoDB, Amazon S3.</p><p><strong>Simple: Lazy Evaluation</strong></p><p>Computation in Spark doesn’t start unless an action is invoked. This is called as Lazy Evaluation and this makes spark faster and resourceful. Spark adds them to DAG (Directed Acyclic Graph) of computation and only when the driver requests some data, does this DAG actually gets executed.</p><p>Spark uses <strong>master/slave architecture</strong>, one master node, and one or many slave worker nodes.</p><p>Here, <strong>Driver</strong> is the central coordinator that runs on master node or name node and executors are on the worker nodes or data nodes that are distributed.</p><p>Spark has its own standalone cluster manager to run the spark applications, it also supports other cluster managers like YARN, Mesos etc.</p><p>Spark uses <strong>RDD</strong> (Resilient Distributed Dataset) which is a fault-tolerant collection of elements that can be operated in parallel. They are immutable in nature.</p><p>RDD’s support two types of operations</p><ul><li><strong>Transformations</strong> : Functions that produces new RDD from existing RDD’s. There are two types of transformations as shown below.</li></ul><ol><li><em>Narrow transformation </em>: doesn’t require the data to be shuffled across the partitions. for example, Map, filter and etc.</li><li><em>Wide transformation</em> : requires the data to be shuffled for example, reduceByKey and Join etc.</li></ol><ul><li><strong>Actions</strong> : Functions that perform some kind of computation over the transformed RDD and sends the computed result from executors to driver.</li></ul><p>Spark translates the RDD transformations into <strong>DAG</strong> and starts the execution. When an action is called the DAG is submitted to the DAG scheduler.</p><p>DAG scheduler divides operators into <strong>Stages</strong> and each Stages are comprised of units of work called as <strong>Tasks</strong>.</p><p>Stages are passed on to the <strong>Task Scheduler</strong>. The task scheduler launches the tasks via cluster manager.</p><p>The <strong>executors</strong> then executes the tasks on the <strong>worker</strong> nodes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/999/1*L1uOdP-ddKkLzHra2W2q6A.jpeg" /><figcaption>Spark execution flow</figcaption></figure><h3>Entry point of Spark</h3><p>After Spark 2.0 the entry point of spark is <strong>Spark Session</strong>. Spark enables its users to create as many sessions as possible for the Spark.</p><p>Prior Spark 2.0, Spark Context was the entry point of any spark application and used to access all spark features and needed a <strong>sparkConf</strong> which had all the cluster configurations and parameters to create a Spark Context object.</p><p>Spark session can be created using the builder pattern.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b298482ed48be1b965b71ced1a4990a2/href">https://medium.com/media/b298482ed48be1b965b71ced1a4990a2/href</a></iframe><p>The spark session <strong>builder</strong> will try to get a spark session if there is one already created (in case of spark shell or databricks ) or create a new one and assigns the newly created SparkSession as the global default.</p><p>Inside the spark session we can get to create the Spark context and create our RDD objects.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d05ef39ec65478d907bd3ec1d73d57f1/href">https://medium.com/media/d05ef39ec65478d907bd3ec1d73d57f1/href</a></iframe><p>Spark gives a straight forward API to create a new session which shares the same spark context. spark.newSession() creates a new spark session object.</p><p>Before implementing Spark session, for Hive or Sql has integrations with spark, seperate SparkContext was needed to be created for each of them. To avoid this Spark Session has been created with well defined API’s for most commonly used components.</p><p>Once we apply transformations to the RDD’s we create an RDD lineage.</p><p>When we apply transformations on an existing RDD it creates a new child RDD, and this <strong>Child RDD carries a pointer to the Parent RDD</strong> along with the metadata about what type of relationship it has with the parent RDD.</p><p>These dependencies are logged as a graph which is called as RDD lineage or RDD dependency graph.</p><p>You can learn about a RDD lineage graph using <strong>RDD.toDebugString</strong> method which gives an output as below.</p><p>We can see the RDD’s created at each transformation for this wordcount example.</p><p><em>HadoopRDD (parentRDD) =&gt; MapPartitionRDD =&gt; MapParitionsRDD(flatMap) =&gt; MapParitionsRDD(Map) =&gt; ShuffledRDD (childRDD)</em></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ea414046987aab9cd33cbb1b8fa1eadf/href">https://medium.com/media/ea414046987aab9cd33cbb1b8fa1eadf/href</a></iframe><p>A lineage will keep track of what all transformations has to be applied on that RDD, including the location from where it has to read the data. It is also called as <strong>Logical execution plan</strong>.</p><p>This RDD lineage is used to <strong>recompute the data if there are any faults</strong> as it contains the pattern of the computation thus the resilience and the fault tolerance of Spark.</p><h3>DAG Scheduler</h3><p>DAG Scheduler is the <strong>scheduling layer</strong> of Apache Spark that implements stage-oriented scheduling. It transforms a logical execution plan to a physical execution plan (using <strong>stages</strong>).</p><p>DAG Scheduler does three things in Spark</p><ul><li>Computes an <strong>execution DAG or Physical execution plan</strong>, i.e. DAG of stages, for a job.</li><li>Determines the preferred locations to run each task on.</li><li>Handles failures due to <strong>shuffle output files</strong> being lost.</li></ul><p>We will cover the how the Physical plan is created in this blog, other two will be discussed in the upcoming blog series.</p><p>DAGScheduler uses event queue architecture to process incoming events, which is implemented by the <strong>DAGSchedulerEventProcessLoop</strong> class. The purpose of the DAGSchedulerEventProcessLoop is to have a separate thread to process events asynchronously and serially, i.e. one by one, and let DAGScheduler do its work on the main thread.</p><p>Whenever an action is called over an RDD, it is submitted as an event of type DAGSchedulerEvent by the Spark Context to DAGScheduler. It is submitted as a JobSubmitted case.</p><p>The first thing done by DAGScheduler is to create a <strong>ResultStage</strong> which will provide the result of the spark job which is submitted.</p><p>Now to execute the submitted job, we need to find out on which operation our RDD is based on. So <strong><em>backtracking</em></strong> begins.</p><p>In backtracking, we find the current operation and the type of RDD it creates. If the current operation produces a <strong>ShuffledRDD</strong> then <strong>shuffle dependency</strong> is detected which creates the <strong>ShuffleMapStage</strong></p><p>DAG scheduler creates a <strong>shuffle boundary</strong> when it encounters Shuffle dependency or Wide transformation and creates a new stage. This new stage’s output will be the input to our ResultStage.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*s6kRyOIJ8rDqJkdrlS35mg.png" /><figcaption>Shuffle boundaries formed at stages where shuffle dependencies are detected</figcaption></figure><p>If we find another shuffle operation happening then again a new shuffleMapStage will be created and will be placed before the current stage (also a ShuffleMapStage) and the newly created shuffleMapStage will provide an input to the current shuffleMapStage.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/393/1*qO0TkBHnRF8F8ADgiRPhDA.png" /><figcaption>Stages scheduled as physical plan</figcaption></figure><p>Hence all the <strong>intermediate stages will be ShuffleMapStages and the last one will always be a ResultStage</strong>.</p><p>The DAG scheduler <strong>pipelines operators together</strong>. Narrow transformations on RDD produces RDD’s without Shuffle dependency that are collected together as a single stage by the DAG scheduler.</p><p>DAG also determines the <strong>execution order</strong> of stages. Execution order is accomplished while building DAG, Spark can understand what part of your pipeline can run in parallel. It optimises minimal stages to run the Job or action.</p><p>The final result of a DAG scheduler is a set of stages and it hands over the stage to <strong>Task Scheduler</strong> for its execution which will do the rest of the computation.</p><h3>Task Scheduler</h3><p>Stages consists of set of tasks which are given as input to the Task scheduler. When a Spark application starts <strong>TaskSchedulerImpl</strong> with a <strong>SchedulerBackend</strong> and DAGScheduler are created and started.</p><p>TaskSchedulerImpl is the default task scheduler in Spark that generates tasks. It uses the SchedulerBackend which schedules tasks on a cluster manager. It also uses the <strong>SchedulableBuilder</strong> to build and manage the <strong>pool</strong> .</p><p>TaskSchedulerImpl submits the tasks using SchedulableBuilder via submitTasks</p><p>submitTasks registers a new TaskSetManager (for the given TaskSet ) and requests the SchedulerBackendend to handle resource allocation offers (from the scheduling system) using reviveOffers method.</p><p>submitTasks requests the SchedulableBuilder to submit the <strong>task</strong> from TaskSetManager to the schedulable pool.</p><p>The schedulable pool can be a</p><ul><li>single flat linked queue (in <strong>FIFO scheduling mode</strong>)</li><li>hierarchy of pools of Schedulables (in <strong>FAIR scheduling mode</strong>)</li></ul><p>Any task either finishes succesfully or fails, TaskSetManager gets notified and also has the power to abort a TaskSet if the number of failures of task is greater than that of spark.task.Maxfailures .</p><h3>Executors</h3><p>Tasks are then scheduled to the acquired <strong>Executors</strong> according to resources and locality constraints. They live in <strong>Worker</strong> <strong>nodes</strong> or <strong>slave</strong> to execute the tasks.</p><p>The ExecutorBackend uses launchTask to send tasks to Executor to get executed where the TaskRunner manages to do the execution of a single Task.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/533/1*bp4t3nOW8vITEzo4qe3z6g.png" /></figure><p>There are two kinds of tasks</p><ul><li><strong>Result task</strong> : computes the result stage and sends result back to the driver.</li><li><strong>ShuffleMap task</strong> : Divides the elements of RDD into multiple buckets based on the partitioner mentioned in Shuffle Dependency.</li></ul><h3>ShuffleMap Task</h3><p>The ShuffleMap Task involves with the <strong>shuffling</strong> of data and the steps involved with it.</p><ul><li>Shuffle write</li><li>Shuffle read</li></ul><p>and the functions that play valuable part in a shuffling process are</p><ol><li><strong><em>Shuffle Manager</em></strong> : Manages the shuffle related components. Invoked within SparkEnv. (after version 1.2 Default : Sort Shuffle Manager)</li><li><strong><em>Shuffle Writer</em></strong> : Handles Shuffle data output logic. Returns <strong>Mapstatus</strong> tracked by <strong>MapOutputTracker</strong> once it writes data into buckets.</li></ol><p>Currently there are three shuffle writers as mentioned below.</p><ul><li>SortShuffleWriter is requested to write records into shuffle partitioned file in disk store.</li><li>BypassMergeSortShuffleWriter is requested to write records into single shuffle block data file.</li><li>UnsafeShuffleWriter is requested to close the internal resources and write out merged spill files.</li></ul><p>3.<strong><em> Shuffle Reader</em></strong> : Fetches data from the buckets.</p><p>4. <strong><em>Shuffle Block Manager</em></strong> : Manages the mapping of data between the buckets and the data blocks written in disk.</p><p>When executed, a ShuffleMapStage saves <strong>map output files</strong> using BlockManager from <strong>Mapper</strong>(MapTask) to buckets via Shuffle Writer that can later be fetched by Shuffle Reader and given to <strong>Reducer</strong>(ReduceTask).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/906/1*1Pq-AyYmKlfwKZiCggQwnA.png" /><figcaption>Shuffle read and Shuffle write</figcaption></figure><p>ShuffleMapStage uses outputLocsand _numAvailableOutputs internal registries to track how many shuffle map outputs are available.</p><p>All intermediate stages will be ShuffleMapStages that are computed will end up with a ResultStage finally that will be computed by TaskRunner and the computed result is sent back to the Driver to display to User.</p><h4>References</h4><p><a href="https://trongkhoanguyen.com/spark/understand-the-scheduler-component-in-spark-core/">Khoa Nguyen Blog</a></p><p><a href="https://jaceklaskowski.gitbooks.io/mastering-apache-spark">Jacek laskowski gitbooks on Spark</a></p><p><a href="https://github.com/apache/spark/tree/master/core/src/main/scala/org/apache/spark">Spark Source code</a></p><p><a href="https://0x0fff.com/spark-architecture-shuffle/">0x000f Blog</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a343b075ad7b" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>