<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Dan Flomin on Medium]]></title>
        <description><![CDATA[Stories by Dan Flomin on Medium]]></description>
        <link>https://medium.com/@flomin-dan?source=rss-d05d57f0980b------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*GlUa0h9J7l4dtud5.jpg</url>
            <title>Stories by Dan Flomin on Medium</title>
            <link>https://medium.com/@flomin-dan?source=rss-d05d57f0980b------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 01:48:34 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@flomin-dan/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[PySpark Planning — A Guide for 1st Timers]]></title>
            <link>https://flomin-dan.medium.com/pyspark-planning-a-guide-for-1st-timers-e9a79a4706aa?source=rss-d05d57f0980b------2</link>
            <guid isPermaLink="false">https://medium.com/p/e9a79a4706aa</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[pyspark]]></category>
            <category><![CDATA[apache-spark]]></category>
            <dc:creator><![CDATA[Dan Flomin]]></dc:creator>
            <pubDate>Wed, 27 Jan 2021 08:06:26 GMT</pubDate>
            <atom:updated>2021-01-27T09:28:52.731Z</atom:updated>
            <content:encoded><![CDATA[<h3>PySpark Planning — A Guide for 1st Timers</h3><h4>Decipher Spark plans and leverage them to your benefit</h4><p>This is the 2nd post in the <strong>PySpark XP</strong> series (<a href="https://flomin-dan.medium.com/pyspark-xp-syntax-f57a9e6a500f">click for the 1st</a>), and its main theme is <strong>Spark plans</strong>.</p><p>In this post you will learn<strong> how to decipher</strong> Spark plans, and learn ways to leverage your new knowledge in order to <strong>optimize Spark jobs</strong> performance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*7dqzPDuw4M4nuUf1" /><figcaption>Photo by <a href="https://unsplash.com/@medbadrc?utm_source=medium&amp;utm_medium=referral">Med Badr Chemmaoui</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h4>Tip #1 —Just try</h4><p>Some Spark developers avoid reading the plan printed by .explain() at all cost.</p><p>I get it, it might be <strong>frightening</strong>.</p><p>One may look online for the meaning of each curse that appears on the plan — I believe it will be beneficial for programming better Spark applications.</p><p>On the other hand, one can just try and play with the plan. I believe it will be a better learning experience.</p><p>Lets try something out —</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/622/1*ELKYtZqUlWI3yTOz1yRVdA.png" /><figcaption>Figure 1</figcaption></figure><p>We created 2 dataframes, each with 1 integer column, which consists of 2M rows with integers up to 2M. We then joined them. The resulting dataframe, df, is made of 2 columns where each row contains the same value twice.</p><p>Let us try and guess what Spark does under the hood —</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/678/1*iHIy84LJHqq5dS_JdQ6HaA.png" /><figcaption>Figure 2</figcaption></figure><p>Just from looking briefly, I see something called <strong>SortMergeJoin</strong>, so this helps understanding the “under the hood” of this join execution.</p><p>Moreover, <strong>I bet you could interpret</strong> most of these lines without a problem.</p><p>Take a minute to do so.</p><p>Now let’s look. For a start —</p><ul><li>Rows where <strong><em>Range</em></strong> appears looks like our use of the .range() function to generate data.</li><li>Rows where <strong><em>Project </em></strong>appears looks like our “select” clause over the generated data using .range(). The first one is our calculation of 3*id % 5 with the column naming, and the second one is just the column naming.</li></ul><p>And now a question — what is <strong><em>Exchange</em></strong>?</p><h4>Tip #2 — Exchange</h4><p>When you decipher Spark plans, you will probably encounter the word <strong><em>Exchange</em></strong>.</p><p>It translates to <strong>shuffling data</strong>.</p><p>Spark moves data from one executor to another, to be able to process the wanted dataframe’s data.</p><p>A common exchange method is <strong><em>hashpartitioning </em></strong>— just as in <strong>Figure 2</strong>.</p><p>Lets observe the lower <em>hashpartitioning</em><strong> </strong>row in the plan.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/647/1*F4LIcBU0IjN0jXK9WE38yw.png" /><figcaption>Figure 3</figcaption></figure><p>This tells us that Spark distributes the data of df2 using a hash over column_2 into 200 different partitions, over our executors.</p><p>More on the magic number 200 on the <strong>4th tip</strong>.</p><h4>Tip #3 —Skewed exchanging</h4><p>What about the first <em>hashpartitioning </em>row? Is it different than the 2nd?</p><p>Lets look —</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*4tqJXKu_-Og0NIayYOVBsg.png" /><figcaption>Figure 4</figcaption></figure><p>I guess it looks the same, give or take.</p><p>Wait a second…</p><p>column_1 holds only 5 values, since we used the <em>modulo</em> operator with 5 over it.</p><p>This means that df1&#39;s data is distributed into <strong>at most 5 partitions ! </strong>At least 195 partitions are empty…</p><p>Lets check that out —</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/688/1*NWnuJouORm1zwo3nL-Jllw.png" /><figcaption>Figure 5</figcaption></figure><p><strong>And I thought Spark is supposed to be highly parallelized…</strong></p><p>This is a common case of a very slow, or rather <strong>stuck</strong>, Spark application. Many times it corresponds to the case we see a stage has 200 tasks, and 199 of them finished already.</p><p>Understanding Spark’s mechanism, and <em>hashpartitioning </em>in particular, helps us diagnose our application, and realize where bottlenecks and low concurrency may occur.</p><p>How will you solve it? Keep reading to the 5th tip to see my suggestion.</p><h4>Tip #4— What is this 200 magic number?</h4><p>This of course it the default value of Spark’s spark.sql.shuffle.partitions.</p><p>This parameter decides how many partitions a shuffling operation will produce.</p><p>For example, <strong>Join </strong>and <strong>groupBy</strong> can cause shuffling. This practically ensures that your Spark application encounters this parameter.</p><p>Therefore its fine tuning can yield enhanced performance of your application.</p><p>Check out <a href="https://github.com/danflomin/medium/blob/main/Tips/Part%202%20-%20Planning/Medium%20-%20Tips%20-%20Planning.ipynb">the notebook</a> to see how to change this parameter.</p><h4>Tip #5 — Solving skewness</h4><p>There are several ways approaching this problem.</p><p>Over some datasets one could leverage prior knowledge to overcome this problem.</p><p>If we fall to the other case:</p><ul><li>In Spark 3 there is a new feature called <strong>adaptive query execution</strong> that “solves” the problem automatically.</li><li>Otherwise, there is a method called <strong>salting </strong>that might solve our problem.</li></ul><p>Many posts were written regarding salting (a reference at the end of this post), which is a cool trick, but not very intuitive at first glance.</p><p>The following is the concept of salting. For a more thorough explanation check out <a href="https://github.com/danflomin/medium/blob/main/Tips/Part%202%20-%20Planning/Medium%20-%20Tips%20-%20Planning.ipynb">the notebook</a> for a code example (same notebook from the 4th tip).</p><ol><li>We concatenate a random value to the key of the “OK” dataframe (not the one that cause the skewed data), df2 in our case. An integer from 1 to 100 for example (this is the salting value).</li><li>Now we would like a join command with df1 to yield the desired result. We will do so by…</li><li>Replicate df1 100 times (the same as the salting value), and add a salt column, with values 1 to 100 (a different value for each row’s replica).</li><li>Join the dataframes with — the salted key in df2 with the concatenation of the key and the salt in df1.</li></ol><p>Voila.</p><h3>Final words</h3><p>Now it is your turn to try and understand Spark plans. Good luck!</p><p>Thanks for reading my 5 tips about <strong>PySpark planning</strong>.</p><h4>Series’ posts</h4><p>(1) <a href="https://medium.com/analytics-vidhya/pyspark-xp-syntax-f57a9e6a500f">PySpark Syntax — 5 Quick Tips</a></p><h4>References</h4><p><a href="https://dzone.com/articles/why-your-spark-apps-are-slow-or-failing-part-ii-da">https://dzone.com/articles/why-your-spark-apps-are-slow-or-failing-part-ii-da</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e9a79a4706aa" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[PySpark XP — Syntax]]></title>
            <link>https://medium.com/analytics-vidhya/pyspark-xp-syntax-f57a9e6a500f?source=rss-d05d57f0980b------2</link>
            <guid isPermaLink="false">https://medium.com/p/f57a9e6a500f</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[spark]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[pyspark]]></category>
            <category><![CDATA[big-data]]></category>
            <dc:creator><![CDATA[Dan Flomin]]></dc:creator>
            <pubDate>Thu, 03 Dec 2020 11:42:51 GMT</pubDate>
            <atom:updated>2021-01-27T09:28:08.969Z</atom:updated>
            <content:encoded><![CDATA[<h3>PySpark Syntax—5 Quick Tips</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tP-dw4Oj_42BYbkdtYbjMA.png" /></figure><p>This is the first post in a series of posts , <strong><em>PySpark XP</em></strong>, each consists of 5 tips. <strong>XP </strong>stands for <strong>experience points</strong>, as the tips are related to matters I learnt from my experience with PySpark. Each post will provide tips about a different aspect of my experience with PySpark.</p><p>The first post is about <strong>syntax</strong>. It will be a valuable lesson for PySpark beginners. More experienced developers can learn from it too (check out <strong>Tip#2</strong>).</p><p>To check this post’s Jupyter notebook click <a href="https://github.com/danflomin/medium/blob/main/Tips/Part%201%20-%20Syntax/Medium%20-%20Tips%20-%20Syntax.ipynb"><strong>here</strong></a>.</p><h4>Tip #1 — The Column object</h4><p>Columns are objects on their own. One can put a Column in a list or another collection, or apply functions on the objects themselves. This disconnects the concept of a Column from its parent dataframe.</p><p>Here is an example.</p><p>We have a dataframe with 2 columns of integers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/214/1*JGeSiiVLNgH2VU1PjkL8oA.png" /><figcaption>Figure 1</figcaption></figure><p>Many, commonly use columns integrally to select or withColumn operations. Like this—</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/632/1*0nLEG_2bC5Kgtuj5h_TIlw.png" /><figcaption>Figure 2</figcaption></figure><p>But the columns can also be multiplied outside the select or withColumn operations.</p><p><strong>Look at the following —</strong></p><p>First we import Spark’s functions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/334/1*8aOaIwwcthQ5tu_EKVaT3w.png" /><figcaption>Figure 3</figcaption></figure><p>Now we put the columns of df in a list.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/643/1*y6jG11lw56G6elSnmvntlw.png" /><figcaption>Figure 4</figcaption></figure><p>Here comes the magic. We multiply each column by 2, and return the multiplied columns “back” to the dataframe.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/546/1*SeW4DY8ToSID1XSKbmNipQ.png" /><figcaption>Figure 5</figcaption></figure><h4>Tip #2 — Select vs. withColumn</h4><p>In my first and last post (click <a href="https://medium.com/p/e273ede0ca13"><strong>here</strong></a> to check it out), I discussed about a Spark join optimization, and how optimization can be done using withColumn instead of join.</p><p>One reader brought to my attention the fact that using withColumn time after time consecutively, should be slower (with respect to driver only) than doing 1 select with all the wanted new columns.</p><p>This is the equivalent to Figure 3, but using withColumn 2 times instead of just 1 select clause.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/820/1*3ORwkwScuw8zHQU003761g.png" /><figcaption>Figure 6</figcaption></figure><p>The reason for this is that a dataframe object is immutable, and therefore after each withColumn, a new dataframe object is returned. The construction of the new dataframe takes time — time consumed by Spark’s internal implementation.</p><p>For a more thorough explanation of the phenomena check the references section at the bottom.</p><h4>Tip #3 — Aliasing</h4><p>A most common case is when one wants to group by and then sum. The easiest syntax leaves us with a column weirdly named (just like on <strong>Tip #1</strong>). Working later with this column name could be frustrating, though not a deal breaker.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/396/1*QJqeswfbrs1WfkiWz53dIQ.png" /><figcaption>Figure 7</figcaption></figure><p>A way for bypassing this problem, would be the following —</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/668/1*LBWtS3WCR3ZpwdWscJ1ldQ.png" /><figcaption>Figure 8</figcaption></figure><h4>Tip #4— selectExpr</h4><p>It is very common in a PySpark code to use a function from pyspark.sql.functions. Importing the needed functions could be a bit of a drag, and one wouldn’t want importing functions all the time (especially in Jupyter notebooks).</p><p>Assume we want to apply the <strong>sin</strong> function over our 2 columns. We could do the following, of course with aliasing (;</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/504/1*wyJbL97vL0VdHGMi8MLc6A.png" /><figcaption>Figure 9</figcaption></figure><p>An alternative would be to give the dataframe a name, and then using the SQL syntax to query the dataframe — like the following example.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/1*Tdi0sv0WdW2Qq0lLOdNyrg.png" /><figcaption>Figure 10</figcaption></figure><p>But, there is an alternative I like better. selectExpr gives the ability to select with SQL syntax over a dataframe, using strings, without the need to name the dataframe or importing the wanted functions.</p><p>Look below to check out how.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/401/1*IVICSvTSl5MlTW4AJCWEAA.png" /><figcaption>Figure 11</figcaption></figure><p>And… we can even alias our columns from within the selectExpr clause!</p><h4>Tip #5 — The Row object</h4><p>A dataframe is on some level, a collection of <strong>Row</strong>s. We are speaking of the class pyspark.sql.Row.</p><p>When you collect a dataframe, you end up with list where each of its elements is a Row. Another place where you encounter Row objects, is when your convert your dataframe into a RDD.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/310/1*G7MqpWWwYNIjSIouvM8utA.png" /><figcaption>Figure 12</figcaption></figure><p>Lets grab a row —</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/305/1*mk03NmWt5zpiHHKbH1TYZw.png" /><figcaption>Figure 13</figcaption></figure><p>One can access a Row’s values in several ways. Here are 2:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/309/1*WNHjeGv7q83vSw3M7CFjJg.png" /><figcaption>Figure 14</figcaption></figure><h3>Conclusion</h3><p>Thanks for reading my 5 tips about <strong>PySpark syntax</strong>.</p><h4>Series’ posts</h4><p>(2) <a href="https://flomin-dan.medium.com/pyspark-planning-a-guide-for-1st-timers-e9a79a4706aa">PySpark Planning — A Guide for 1st Timers</a></p><h4>References</h4><p><a href="https://medium.com/@manuzhang/the-hidden-cost-of-spark-withcolumn-8ffea517c015">Tip #3</a> — “The hidden cost of Spark <em>withColumn</em>”.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f57a9e6a500f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/pyspark-xp-syntax-f57a9e6a500f">PySpark XP — Syntax</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Optimizing Your Spark Joins in an Unfashionable Manner]]></title>
            <link>https://flomin-dan.medium.com/optimizing-your-spark-joins-in-an-unfashionable-manner-e273ede0ca13?source=rss-d05d57f0980b------2</link>
            <guid isPermaLink="false">https://medium.com/p/e273ede0ca13</guid>
            <category><![CDATA[spark]]></category>
            <category><![CDATA[pyspark]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[optimization]]></category>
            <category><![CDATA[big-data]]></category>
            <dc:creator><![CDATA[Dan Flomin]]></dc:creator>
            <pubDate>Sat, 31 Oct 2020 19:02:06 GMT</pubDate>
            <atom:updated>2020-11-04T22:04:21.945Z</atom:updated>
            <content:encoded><![CDATA[<h3>Optimize Spark Joins Unfashionably</h3><p>TL;DR —I optimized Spark joins and reduced runtime from <strong>90 mins</strong> to just <strong>7 mins</strong>. Use a <em>withColumn </em>operation instead of a <em>join</em> operation and optimize your Spark joins <strong>~10 times faster</strong>.</p><p>If you are an experienced Spark developer, you have probably encountered the pain in joining dataframes. It is like you must be a true master to be able to join dataframes efficiently. The questions one may ask himself are:</p><ul><li>Should I repartition my dataframe?</li><li>Should I broadcast the smaller dataframe?</li><li>What about the <em>spark.shuffle.partitions </em>parameter?</li></ul><p>These questions have been occupying many people for a long time. Some really nice rules of thumb were proposed over the years. A good rule of thumb and intuition help a lot in discovering what is the best method to perform a join, but nothing beats trial and error in my opinion.</p><p>I am about to present a really cool concept I developed while suffering tremendously from poor join performance.</p><h4>What was I facing?</h4><p>My primary data was a table with 300 columns and 200 million rows. About 100 columns were coded, and needed to be deciphered. We will call it <strong><em>DATA</em></strong>.</p><p>Luckily, I have another table, with translations to these codes. This table is much smaller — only 300k rows with a small number of columns. We will call it <strong><em>MAPPING</em></strong>.</p><p><em>MAPPING </em>consists of one-to-one translations. For each value needed to be deciphered, there is only 1 value available as translation, with respect to its <strong><em>context</em></strong>. The <em>context</em> directs us on which translation of a code is to be used.</p><p>The naïve approach is using the Spark SQL API. We would iterate over the coded columns in <em>DATA</em>, and for each of those, we would find a translation in <em>MAPPING.</em></p><p>More advanced approaches would have us playing with parameters (such as <em>spark.shuffle.partitions</em>) or trying to broadcast\repartition dataframes.</p><p>Either case, I was facing 100 join operations, and I didn’t like it.</p><p>Surprisingly— performing the naïve approach in my project <strong>took forever, </strong>about<strong> 90 minutes</strong>. <br>Keep scrolling and check out how I reduced it to <strong>7 minutes</strong>.</p><h4>Lets demonstrate</h4><p>In this example, we are going to handle <strong>inventory data</strong>. We have several departments — hence a column named <em>department_id</em>. Each department has its own products — each of those products have <em>product_id</em> and <em>product_name </em>columns.</p><p>Here, the <em>context</em> is <em>department_id</em>, the coded column is <em>product_id</em>, and its translation is <em>product_name</em>.</p><p>Below is how <em>MAPPING </em>would look like.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/41888e10e055cc4d0d022c62ddee8015/href">https://medium.com/media/41888e10e055cc4d0d022c62ddee8015/href</a></iframe><p>Now, we have a very large DB consists of orders of inventory items. DATA would look like this.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ce51d9e2aeb40d0612521ea901cf061b/href">https://medium.com/media/ce51d9e2aeb40d0612521ea901cf061b/href</a></iframe><p>All we would like to do in our processing, is to add the <em>product_name</em> column from <em>MAPPING</em>, to our data in <em>DATA</em>.</p><p>Now, lets have a look on how the naïve approach would look like:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*b50-OmqJiq-SaW8Ryy7Klg.png" /><figcaption>Figure 1</figcaption></figure><p>In this example, we have only 1 column to be joined. In my use case, there are <strong>200M rows</strong> and <strong>100 columns </strong>of <em>product_id —</em> and each column needs to be joined on!</p><p>Usually a join will consume lots of networking for each join operation. Moreover, hashing\sorting will occur in order to complete a join operation. Anyway, we are talking about time <strong>expensive </strong>operations that <strong>slow down</strong> our processing.</p><h4>Collect() is not a bad word</h4><p>Working with databases helped me realize what I really wanted to have in my Spark application. I wanted an <strong>indexed join</strong>. Furthermore, <em>DATA</em> contained only a small number of values in the <em>context</em> column — leading me to think of <strong>partitioned </strong>tables.</p><p>I didn’t want spark to shuffle every time I joined these dataframes, I didn’t want spark to perform sorting for every column I tried to translate.</p><p>I literally pictured in my head how each dataframe row access a single DB partition, then uses the DB’s index, and retrieves its translation.</p><p>This led me to experiment joining dataframes with a dictionary.</p><p><strong>What does it even mean?</strong></p><p>Instead of joining 2 dataframes for 100 hundred times, I turned the join operation into a <strong><em>withColumn</em> </strong>operation.</p><p>Let me remind you that <em>MAPPING </em>is a one-to-one translation. Leveraging this fact, we can create a <em>user-defined-function </em>(udf)<em> </em>that maps the coded value into a deciphered value.</p><p>Below you can see how to perform a <em>withColumn</em> join. <br>Later on I will discuss the <strong>obvious advantages</strong> using this kind of join.</p><p>First, we collect <em>mapping_df </em>into a hierarchical dictionary — the first hierarchy is <em>department_id</em>, and the second is <em>product_id</em>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/938/1*MnXHhn-9XXWflL_52NzgIQ.png" /><figcaption>Figure 2</figcaption></figure><p>Then, we broadcast the dictionary, create a <em>udf </em>using the broadcasted dictionary, and join the values using the <em>withColumn </em>operation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CaQNteqZ3KnBZL_VdNQ1ZQ.png" /><figcaption>Figure 3</figcaption></figure><h4>Now is the time to get in panic</h4><p>Yes. I bet you could optimize your Spark pipeline’s join operations <strong>manyfold</strong>. I will now show a simple experiment proving this method to be of great value.</p><p>For the <strong>full experiment</strong> notebook, <strong>click </strong><a href="https://github.com/danflomin/medium/blob/main/SparkJoinOptimization/optimizing_your_spark_joins_in_an_unfashionable_manner.ipynb"><strong>here</strong></a>.</p><p>First, I created mapping_df, which contains 300K rows. We have 15 departments and 20K products per department.</p><p>Then, I created 2 dataframes — one for each join method, <strong>grouped by</strong> some columns, and executed the <em>.count()</em> action. The data in both dataframes is the same (both logically and practically — I validated it of course).<br>I ran the <em>.count()</em> action several times over each dataframe to get an average of the execution time.</p><p>In the following experiment, I computed for the average run time for 5 runs. Here are some of my findings —</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/919/1*-1s4Mfjfx4fvdihRZlOpPA.png" /><figcaption>Figure 4</figcaption></figure><ol><li>For a dataframe of 100K rows, we got better results using a <em>withColumn</em> join by up to <strong>8.9 times faster </strong>than the naïve approach.</li><li>The higher the number of <em>product_id</em> columns to join on, the <strong>greater </strong>the <strong>relative difference</strong> between the executions was.</li><li>When we increased the number of rows (1M, 3M, 10M, 50M), and fixed the number of columns to join on (10), the relative difference between the run times decreased, as seen below.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/941/1*_QyH31KSa7KENymZdUJliw.png" /><figcaption>Figure 5</figcaption></figure><p><strong>“What about broadcast joins???”</strong></p><p>That is a great question. Shouldn’t a regular broadcast join work better? It doesn’t involve UDFs and all the serialization problems. Looks like it should perform at least as good as a <em>withColumn</em> join.</p><p>You are absolutely right. Broadcast joins can work very good. In fact, they perform much better better when <em>mapping_df </em>is broadcasted first, and then used in the join operation to create <em>sql_df</em>.</p><p>But there is a but… :)</p><p>Below you can see the results for the experiment when <em>mapping_df</em> is broadcasted. It makes the action over <em>sql_df</em> much faster— 23 seconds without broadcasting for 10 columns <strong>compared to </strong>12 seconds with broadcasting for 20 columns.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/946/1*YPD8v78LgovdoDdyYdg8eQ.png" /><figcaption>Figure 6</figcaption></figure><p>But what we do see, is that when the number of columns continues to increase, <strong>the <em>withColumn</em> approach takes the lead by far</strong>.</p><h4>Our plans finally explained</h4><p>Another major advantage of using an <em>indexed join</em>, is how the final execution plan looks like. It turns a plan which is un-understandable, into a plan which a novice Spark developer can handle.</p><p>Since a <em>withColumn</em> operation is somewhat equivalent to a <em>select </em>clause, Spark groups them all on the same stage, and therefore, all these “join” operations are executed in parallel.</p><p>On the other hand, since a <em>join </em>operation require a stage for itself, the joins are executed one after the other.</p><p>On the <em>.explain()</em> string, the <em>withColumn </em>operations appear on the same line, and doesn’t require additional stages to be executed (such as “<em>Exchange </em>“ — shuffling).<br>On the other hand, each join operation requires preliminary work — shuffling the data, sorting it, and only then joins it (for SortMergeJoin). This turns a simple join operation into a headache of a plan.</p><p>In extreme cases, such as when the number of columns is very large, Spark’s stack size could be overflowed and crash the application. This is due to a too long plan.</p><h3>Conclusion</h3><p>The <em>withColumn </em>approach worked very good for me, but it isn’t a magic trick one could apply to solve all problems.</p><p>When the number of columns to join is very large, this method could be very useful — but there is a tradeoff between the number of rows and number of columns. Be aware!</p><p>Magic tricks and rules of thumb can’t beat trial and error.</p><p>For me it works like a feedback loop.</p><ol><li>Experimenting leads me to: better understanding and intuition, and greater knowledge.</li><li>My intuition lead me to propose such a method to solve my poor join performance.</li><li>Experimenting my intuition got me back to (1).</li></ol><p>Thanks for reading.<br>Please comment and share use cases this method helped you with.</p><p><strong>Acknowledgments </strong>— <a href="https://medium.com/u/2af40fba0fe5">Yaniv Harpaz</a>, for his great effort of helping me with Azure, for advising me when things weren’t going the way I thought they would, and for urging me to keep writing this article.</p><h4>Footnote — Experiments Workspace</h4><p>I worked with:</p><ul><li>Jupyter notebooks</li><li>Azure’s HDInsight 4.0 (2 master nodes, 8 worker nodes — 16 cores and 112 GB RAM each)</li><li>Pyspark — Spark 2.4 (2.4.5.4.1.1.2)</li><li>Python 3.5.2</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e273ede0ca13" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>