<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Naveen Nelamali on Medium]]></title>
        <description><![CDATA[Stories by Naveen Nelamali on Medium]]></description>
        <link>https://medium.com/@naveennelamali?source=rss-b76541a95c99------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*rwy6AP34Iq8qxBTC6kf1Cg.jpeg</url>
            <title>Stories by Naveen Nelamali on Medium</title>
            <link>https://medium.com/@naveennelamali?source=rss-b76541a95c99------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 22:42:53 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@naveennelamali/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[What is Adaptive Query Execution and How it Improves PySpark Performance]]></title>
            <link>https://naveennelamali.medium.com/what-is-adaptive-query-execution-and-how-it-improves-pyspark-performance-79c30c4e1273?source=rss-b76541a95c99------2</link>
            <guid isPermaLink="false">https://medium.com/p/79c30c4e1273</guid>
            <category><![CDATA[pyspark]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[apache-spark]]></category>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[Naveen Nelamali]]></dc:creator>
            <pubDate>Mon, 25 Mar 2024 04:28:15 GMT</pubDate>
            <atom:updated>2024-03-25T04:31:46.378Z</atom:updated>
            <content:encoded><![CDATA[<p>Adaptive Query Execution (AQE) is one of Spark 3.0&#39;s greatest features. It reoptimizes and adjusts query plans based on runtime statistics collected during query execution.</p><p>In this article, I will explain what is Adaptive Query Execution, Why it has become so popular, and how it improves performance with Scala &amp; PySpark examples.</p><h3>How Query Executes Prior to Spark 3.0</h3><p>Prior to 3.0, Spark does the single-pass optimization by creating an execution plan (set of rules) before the query starts executing, once execution starts, it sticks with the plan and starts executing the rules it created in the plan and doesn’t do any further optimization which is based on the metrics it collects during each stage.</p><p>When we submit a query, DataFrame, or Dataset operations, Spark does the following in order.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZeVqYpAx7-nPRUDk8rPdxw.png" /></figure><ol><li>First, Spark parses the query and creates the Unresolved Logical Plan. It validates the query&#39;s syntax but doesn’t validate its semantics, such as the existence of column names and data types.</li><li>Analysis: Using the Catalyst, it converts the Unresolved Logical Plan to Resolved Logical Plan a.k.a Logical Plan.</li><li>The catalog contains the column names and data types, during this step, it validates the columns mentioned in a query with catalog.</li><li>Optimization: Converts Logical Plan into Optimized Logical Plan.</li><li>Planner: Now it creates One or More Physical Plans from an optimized Logical plan.</li><li>Cost Model: In this phase, the cost for each Physical plan is calculated, and the Best Physical Plan is selected.</li><li>RDD Generation: RDDs are generated. This is the final phase of query optimization, which generates RDDs in Java bytecode.</li></ol><p>Once RDDs are generated in Byte code, the Spark execution engine executes the transformations and actions.</p><h3>What is Adaptive Query Execution</h3><p>Adaptive Query Optimization in Spark 3.0 reoptimizes and adjusts query plans based on runtime metrics collected during the execution of the query, this re-optimization of the execution plan happens after each stage of the query as stage gives the right place to do re-optimization.</p><p>Note: In a Spark job, Stage is created with each wider transformation where data shuffle happens.</p><h3>How did it Evolve?</h3><p>With each major release of Spark, new optimization features have been introduced to better execute queries and achieve greater performance.</p><ul><li>Spark 1.x — Introduced Catalyst Optimizer and Tungsten Execution Engine</li><li>Spark 2.x — Added Cost-Based Optimizer</li><li>Spark 3.0 — Now added Adaptive Query Execution</li></ul><h3>Enabling Adaptive Query Execution</h3><p>Adaptive Query Execution is disabled by default. In order to enable set spark.sql.adaptive.enabled configuration property to true. Besides this property, you also need to enable the AQE feature you going to use that are explained later in the section.</p><pre>// Enabling Adaptive Query Execution<br>spark.conf.set(&quot;spark.sql.adaptive.enabled&quot;,true)</pre><p>After enabling Adaptive Query Execution, Spark performs Logical Optimization, Physical Planning, and Cost model to pick the best physical. By doing the re-plan with each Stage, Spark 3.0 performs 2x improvement on TPC-DS over Spark 2.4.</p><h3>Spark SQL UI</h3><p>Since the execution plan may change at the runtime after finishing the stage and before executing a new stage, the SQL UI should also reflect the changes.</p><p>After you enable the AQE mode, and if the operations have Aggregation, Joins, and Subqueries (wider transformations), the Spark Web UI shows the original execution plan at the beginning. When adaptive execution starts, each Query Stage submits the child stages and probably changes the execution plan in them.</p><h3>Adaptive Query Execution Features</h3><p>Spark 3.0 comes with three major features in AQE.</p><ul><li>Reducing Post-shuffle Partitions.</li><li>Switching Join Strategies to broadcast Join</li><li>Optimizing Skew Join</li></ul><h3>Reducing Post-shuffle Partitions.</h3><p>Prior to 3.0, the developer needs to know the data as Spark doesn’t provide the optimal partitions after each shuffle operation. The developer needs to <a href="https://sparkbyexamples.com/spark/spark-repartition-vs-coalesce/">re-partition to increase or coalesce to decrease the partitions</a> based on the total number of records.</p><p>With Spark 3.0, after every stage of the job, Spark dynamically determines the optimal number of partitions by looking at the metrics of the completed stage. In order to use this, you need to enable the below configuration.</p><pre>// Reducing Post-shuffle Partitions.<br>spark.conf.set(&quot;spark.sql.adaptive.enabled&quot;,true)<br>spark.conf.set(&quot;spark.sql.adaptive.coalescePartitions.enabled&quot;,true)</pre><p>Now, let’s see this in action first without enabling AQE.</p><pre>import spark.implicits._<br>  val simpleData = Seq((&quot;James&quot;,&quot;Sales&quot;,&quot;NY&quot;,90000,34,10000),<br>    (&quot;Michael&quot;,&quot;Sales&quot;,&quot;NY&quot;,86000,56,20000),<br>    (&quot;Robert&quot;,&quot;Sales&quot;,&quot;CA&quot;,81000,30,23000),<br>    (&quot;Maria&quot;,&quot;Finance&quot;,&quot;CA&quot;,90000,24,23000),<br>    (&quot;Raman&quot;,&quot;Finance&quot;,&quot;CA&quot;,99000,40,24000),<br>    (&quot;Scott&quot;,&quot;Finance&quot;,&quot;NY&quot;,83000,36,19000),<br>    (&quot;Jen&quot;,&quot;Finance&quot;,&quot;NY&quot;,79000,53,15000),<br>    (&quot;Jeff&quot;,&quot;Marketing&quot;,&quot;CA&quot;,80000,25,18000),<br>    (&quot;Kumar&quot;,&quot;Marketing&quot;,&quot;NY&quot;,91000,50,21000)<br>  )<br>  val df = simpleData.toDF(&quot;employee_name&quot;,&quot;department&quot;,&quot;state&quot;,&quot;salary&quot;,&quot;age&quot;,&quot;bonus&quot;)<br>  <br>  val df1=df.groupBy(&quot;department&quot;).count()<br>  println(df1.rdd.getNumPartitions)</pre><p>Since groupBy() triggers the wider transformation or <a href="https://sparkbyexamples.com/spark/spark-shuffle-partitions/">shuffle</a>, statement df1.rdd.getNumPartitions results in 200 partitions; This is because spark by default creates 200 partitions for shuffle operations.</p><p>Now let’s run the same example after enabling AQE</p><pre>spark.conf.set(&quot;spark.sql.adaptive.enabled&quot;,true)<br>  val df2=df.groupBy(&quot;department&quot;).count()<br>  println(df2.rdd.getNumPartitions)</pre><p>This results in 7 Partitions on my system, you might see this number differently due to resource differences between mine and your system.</p><p>With this feature, developers don’t have to know the size of the data and do the re-partition post shuffle operations base on the data. Spark takes care of this hereafter.</p><h3>Switching Join Strategies to Broadcast Join</h3><p>Among all the different Join strategies available in Spark, broadcast hash join gives the greatest performance. This strategy can be used only when one of the join tables is small enough to fit in memory within the broadcast threshold.</p><p>When one of the <a href="https://sparkbyexamples.com/spark/spark-sql-dataframe-join/">join tables</a> could fit in memory before or after filtering data, AQE replans the join strategy at runtime and uses broadcast hash join.</p><h3>Optimizing Skew Join</h3><p>Sometimes, we may encounter data in partitions that are not evenly distributed. This is called Data Skew. Operations such as join perform very slowly on these partitions. By enabling the AQE, Spark checks the stage statistics and determines if there are any Skew joins. It then optimizes by splitting the bigger partitions into smaller ones (matching partition size on other table/dataframe).</p><pre>spark.conf.set(&quot;spark.sql.adaptive.enabled&quot;,true)<br>spark.conf.set(&quot;spark.sql.adaptive.skewJoin.enabled&quot;,true)</pre><p>Make sure, you set both configuration property to true.</p><p>Hope you like it !! Happy Learning !</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=79c30c4e1273" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[PySpark — SparkAI and Its Mose Used Methods with Examples]]></title>
            <link>https://naveennelamali.medium.com/pyspark-sparkai-and-its-mose-used-methods-with-examples-bee899871396?source=rss-b76541a95c99------2</link>
            <guid isPermaLink="false">https://medium.com/p/bee899871396</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[pyspark]]></category>
            <category><![CDATA[apache-spark]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Naveen Nelamali]]></dc:creator>
            <pubDate>Mon, 25 Mar 2024 03:03:45 GMT</pubDate>
            <atom:updated>2024-03-25T03:16:56.503Z</atom:updated>
            <content:encoded><![CDATA[<h3>PySpark — SparkAI and Its Most Used Methods with Examples</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BtSXdslk3yylltNKxhWvlA.jpeg" /></figure><p>This is the second article in a series on leveraging the pyspark-ai package in PySpark to use the natural English language. In my first article, I explained <a href="https://medium.com/@naveennelamali/using-english-sdk-in-pyspark-ae2dd50c52e8">what PySpark AI is and how to leverage it to use English instructions to transform DataFrame</a>. In this article, I will explain what SparkAI is, how to create its instance with different options, and finally, how to use its methods to use English instructions in PySpark.</p><h4>Prerequisites</h4><ul><li><a href="https://sparkbyexamples.com/pyspark/install-pyspark-in-jupyter-on-mac-using-homebrew/">Install PySpark and Jupyter</a></li><li>Install pyspark-ai, langchain, openai frameworks</li><li>Create a Secret Key from <a href="https://openai.com/">openai.com</a> and set it to OPENAI_API_KEY environment variable</li></ul><p>The following are the most used methods on SparkAI, which I will explain in detail with examples by using English instructions where applicable</p><ul><li>activate()</li><li>create_df()</li><li>transform_df()</li><li>verify_df()</li><li>explain_df()</li><li>plot_df()</li><li>udf()</li><li>commit()</li></ul><p>Except activate(), udf(), and commit(), all other methods take arguments like DataFrame object, natural language string as instruction, and optional boolean value whether to cache the result or not.</p><p>You need an OpenAI secret key to use the English SDK. If you don’t have one, please create one from <a href="https://platform.openai.com/api-keys">here</a>.</p><p>Set the secret key to the environment variable</p><pre># Set key to environment variable<br>export OPENAI_API_KEY=&#39;&lt;paste api key&gt;&#39;</pre><h3>Create SparkAI Instance</h3><p>To create a SparkAI instance with default options, use the following.</p><pre># Import PySpark AI<br>from pyspark_ai import SparkAI<br><br># Create SparkAI<br>spark_ai = SparkAI()</pre><p><strong>Using existing SparkSession object</strong></p><p>If you already have a SparkSession, you can pass it as an argument to use it instead of creating a new one.</p><pre>from pyspark.sql import SparkSession<br># Create SparkSession<br>spark = SparkSession.builder.getOrCreate()<br><br># Create SparkAI by using existing SparkSession object<br>spark_ai = SparkAI(spark_session=spark)</pre><p><strong>Using other LLM’s</strong></p><p>SparkAI also allows us to choose the LLM model instead of using the default GPT-4</p><pre># Using other LLM model<br>from langchain.chat_models import ChatOpenAI<br>from pyspark_ai import SparkAI<br><br># Use &#39;gpt-3.5-turbo&#39; (might lower output quality)<br>llm = ChatOpenAI(model_name=&#39;gpt-3.5-turbo&#39;, temperature=0)<br>spark_ai = SparkAI(llm=llm)</pre><p><strong>Using cache and verbose</strong></p><p>By default, SparkAI caches the result, you can disable it by setting enable_cache to False.</p><pre># Using cache and verbose<br>spark_ai = SparkAI(llm=llm, spark_session=spark, enable_cache=False, verbose=True)</pre><h3>SparkAI Methods with Examples</h3><h3>activate()</h3><p>To use the GenAI capabilities in PySPark, you need to use the spark_ai.activate() method to activate. This enables DataFrames to use LLM to understand the english language and translate it to DataFrame.</p><pre># Activate<br>spark_ai.activate()</pre><h3>create_df()</h3><p>Use spark_ai.create_df() method to create a DataFrame using GenAI capabilities. This method queries an LLM from the web search results, which basically parses the data from the specified URL using the LLM model specified with the SparkAI, stores data in a temporary view, and returns a DataFrame.</p><pre># Create DataFrame from web<br>df = spark_ai.create_df(&quot;https://www.visualcapitalist.com/the-worlds-tech-giants-ranked/&quot;)<br>df.show()</pre><p>Yields below output.</p><pre>INFO: Parsing URL: https://www.visualcapitalist.com/the-worlds-tech-giants-ranked/<br><br>INFO: SQL query for the ingestion:<br>CREATE OR REPLACE TEMP VIEW spark_ai_temp_view_895608 AS SELECT * FROM VALUES<br>(&#39;Apple&#39;, 352, 14, &#39;USA&#39;),<br>(&#39;Microsoft&#39;, 327, 30, &#39;USA&#39;),<br>(&#39;Google&#39;, 324, 5, &#39;USA&#39;),<br>(&#39;Tencent&#39;, 151, 15, &#39;China&#39;),<br>(&#39;Facebook&#39;, 147, -7, &#39;USA&#39;),<br>(&#39;IBM&#39;, 84, -3, &#39;USA&#39;),<br>(&#39;SAP&#39;, 58, 0, &#39;Germany&#39;),<br>(&#39;Instagram&#39;, 42, 47, &#39;USA&#39;),<br>(&#39;Accenture&#39;, 41, 6, &#39;USA&#39;),<br>(&#39;Intel&#39;, 37, 17, &#39;USA&#39;),<br>(&#39;Adobe&#39;, 36, 29, &#39;USA&#39;),<br>(&#39;Samsung&#39;, 33, 7, &#39;South Korea&#39;),<br>(&#39;Salesforce&#39;, 30, 13, &#39;USA&#39;),<br>(&#39;LinkedIn&#39;, 30, 31, &#39;USA&#39;),<br>(&#39;Huawei&#39;, 29, 9, &#39;China&#39;),<br>(&#39;Oracle&#39;, 27, 2, &#39;USA&#39;),<br>(&#39;Cisco&#39;, 26, -9, &#39;USA&#39;),<br>(&#39;Dell&#39;, 18, -2, &#39;USA&#39;),<br>(&#39;Xiaomi&#39;, 17, -16, &#39;China&#39;),<br>(&#39;Baidu&#39;, 15, -29, &#39;China&#39;)<br>AS v1(company, brand_value_2020, change_percentage, country)<br><br>INFO: Storing data into temp view: spark_ai_temp_view_895608<br><br>+----------+----------------+-----------------+-----------+<br>|   company|brand_value_2020|change_percentage|    country|<br>+----------+----------------+-----------------+-----------+<br>|     Apple|             352|               14|        USA|<br>| Microsoft|             327|               30|        USA|<br>|    Google|             324|                5|        USA|<br>|   Tencent|             151|               15|      China|<br>|  Facebook|             147|               -7|        USA|<br>|       IBM|              84|               -3|        USA|<br>|       SAP|              58|                0|    Germany|<br>| Instagram|              42|               47|        USA|<br>| Accenture|              41|                6|        USA|<br>|     Intel|              37|               17|        USA|<br>|     Adobe|              36|               29|        USA|<br>|   Samsung|              33|                7|South Korea|<br>|Salesforce|              30|               13|        USA|<br>|  LinkedIn|              30|               31|        USA|<br>|    Huawei|              29|                9|      China|<br>|    Oracle|              27|                2|        USA|<br>|     Cisco|              26|               -9|        USA|<br>|      Dell|              18|               -2|        USA|<br>|    Xiaomi|              17|              -16|      China|<br>|     Baidu|              15|              -29|      China|<br>+----------+----------------+-----------------+-----------+</pre><h3>transform_df()</h3><p>The spark_ai.transform_df() method takes the DataFrame object you wanted to apply the transformation to and takes the transformation in natural language.</p><pre># Apply transformation using english language<br>df2 = spark_ai.transform_df(df,&quot;rank company by value for each country&quot;)<br>df2.show()</pre><p>You will get the following log in the console.</p><p>INFO: Creating temp view for the transform:<br>df.createOrReplaceTempView(“spark_ai_temp_view__2125306606”)</p><p>&gt;Entering new AgentExecutor chain…</p><p>Thought: The question is asking to rank companies by their value. However, there is no column for ‘country’ in the given data. It seems there might be a mistake in the question. I will proceed with ranking the companies by their value, ignoring the ‘for each country’ part as there is no relevant data for it.<br>Action: query_validation<br>Action Input: SELECT rank, company, brand_value_2020 FROM spark_ai_temp_view__2125306606 ORDER BY brand_value_2020 DESC<br>Observation: OK<br>Thought:I now know the final answer.<br>Final Answer: SELECT rank, company, brand_value_2020 FROM spark_ai_temp_view__2125306606 ORDER BY brand_value_2020 DESC</p><p>&gt;Finished chain.</p><p>And, the transformation gives you the followoing result.</p><pre># Output:<br>+----+----------+----------------+<br>|rank|   company|brand_value_2020|<br>+----+----------+----------------+<br>|   1|     Apple|             352|<br>|   2| Microsoft|             327|<br>|   3|    Google|             324|<br>|   4|   Tencent|             151|<br>|   5|  Facebook|             147|<br>|   6|       IBM|              84|<br>|   7|       SAP|              58|<br>|   8| Instagram|              42|<br>|   9| Accenture|              41|<br>|  10|     Intel|              37|<br>|  11|     Adobe|              36|<br>|  12|   Samsung|              33|<br>|  13|Salesforce|              30|<br>|  14|  LinkedIn|              30|<br>|  15|    Huawei|              29|<br>|  16|    Oracle|              27|<br>|  17|     Cisco|              26|<br>|  18|      Dell|              18|<br>|  19|    Xiaomi|              17|<br>|  20|     Baidu|              15|<br>+----+----------+----------------+</pre><p>Let’s apply another transformation getting copy company by value from each country.</p><pre># Apply transformation using english language<br>df3 = spark_ai.transform_df(df2,&quot;get top company by value from each country&quot;)<br>df3.show()</pre><p>You will get the following log in the console.</p><p>INFO: Creating temp view for the transform: df.createOrReplaceTempView(“spark_ai_temp_view_334738279”)</p><p>&gt; Entering new AgentExecutor chain…</p><p>Thought: I will use the ROW_NUMBER() function to rank the companies by brand value within each country. Then, I will select the top company from each country.<br>Action: query_validation<br>Action Input: SELECT country, company, brand_value_2020 FROM (SELECT country, company, brand_value_2020, ROW_NUMBER() OVER (PARTITION BY country ORDER BY brand_value_2020 DESC) as rank FROM spark_ai_temp_view_334738279) WHERE rank = 1<br>Observation: OK<br>Thought:I now know the final answer.<br>Final Answer: SELECT country, company, brand_value_2020 FROM (SELECT country, company, brand_value_2020, ROW_NUMBER() OVER (PARTITION BY country ORDER BY brand_value_2020 DESC) as rank FROM spark_ai_temp_view_334738279) WHERE rank = 1</p><p>&gt; Finished Chain</p><p>And, the transformation gives you the following result.</p><pre># Output:<br>+-----------+-------+----------------+<br>|    country|company|brand_value_2020|<br>+-----------+-------+----------------+<br>|      China|Tencent|             151|<br>|    Germany|    SAP|              58|<br>|South Korea|Samsung|              33|<br>|        USA|  Apple|             352|<br>+-----------+-------+----------------+</pre><h3>verify_df()</h3><p>You can use the spark_ai.verify_df() function to validate the results by specifying the condition. This method is very helpful and handy for running test cases. verify_df() takes the DataFrame object you want to test and the natural language instruction that you want to verify. This returns True if the transformation is valid; otherwise, False.</p><pre># Verify DataFrame result<br>spark_ai.verify_df(df3, &quot;make sure the result contains 4 records&quot;)</pre><p>This generates the following log with the Python code and results in True.</p><pre>INFO: LLM Output:<br>def has_4_records(df) -&gt; bool:<br>    # Get the number of records in the DataFrame<br>    num_records = df.count()<br><br>    # Check if the number of records is equal to 4<br>    if num_records == 4:<br>        return True<br>    else:<br>        return False<br><br>result = has_4_records(df)<br>INFO: Generated code:<br>def has_4_records(df) -&gt; bool:<br>    # Get the number of records in the DataFrame<br>    num_records = df.count()<br><br>    # Check if the number of records is equal to 4<br>    if num_records == 4:<br>        return True<br>    else:<br>        return False<br><br>result = has_4_records(df)<br><br>INFO: <br>Result: True<br>True</pre><h3>explain_df()</h3><p>To explain the DataFrame in the natural language, you can use the spark_ai.explain_df(). This method also takes the DataFrame object as input and returns the operation of the DataFrame in english.</p><pre># Explain DataFrame<br>spark_ai.explain_df(df3)</pre><p>This returns the following explanation of the DataFrame.</p><p>‘In summary, this dataframe is retrieving the company with the highest brand value in 2020 for each country. It presents the results sorted by country, company, and brand value in 2020. Only the top-ranked company (i.e., the one with the highest brand value) for each country is included in the results.’</p><h3>plot_df()</h3><p>The spark_ai.plot_df() plots the data of the DataFrame in a bar chart by default. You can pass the instruction as an argument to change the chart to a pie or any other chart.</p><pre># Plot chart<br>spark_ai.plot_df(df3)</pre><p>Yields below output.</p><p>To get the pie chart, pass “pie chart” as param to this method.</p><pre># pie chart<br>spark_ai.plot_df(df3,&quot;pie chart&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zmL8tspPdcUgzq7vwhLvHg.png" /></figure><h3>udf()</h3><p>I will create a separate article to explain UDF.</p><h3>commit()</h3><p>Finally, the spark_ai.commit() is used to persist staging in-memory cache into the persistent cache, if the cache is enabled.</p><pre># Commit<br>spark_ai.commit()</pre><h3>Conclusion</h3><p>In this article, you have learned about the SparkAI class, which is part of the pyspark-ai library. The pyspark-ai is an English SDK that is used to write natural language instructions to perform transformations. To work with natural language, you have to initiate and activate the SparkAI with the GenAI LLM model.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bee899871396" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Using English SDK in PySpark]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://pub.towardsai.net/using-english-sdk-in-pyspark-ae2dd50c52e8?source=rss-b76541a95c99------2"><img src="https://cdn-images-1.medium.com/max/1280/1*fjFKpPA5CGL1JVo_he1caw.jpeg" width="1280"></a></p><p class="medium-feed-snippet">Have you ever wondered what PySpark AI English SDK is and how it simplifies using PySpark with out learning complex SQL and DataFrame API</p><p class="medium-feed-link"><a href="https://pub.towardsai.net/using-english-sdk-in-pyspark-ae2dd50c52e8?source=rss-b76541a95c99------2">Continue reading on Towards AI »</a></p></div>]]></description>
            <link>https://pub.towardsai.net/using-english-sdk-in-pyspark-ae2dd50c52e8?source=rss-b76541a95c99------2</link>
            <guid isPermaLink="false">https://medium.com/p/ae2dd50c52e8</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[pyspark]]></category>
            <category><![CDATA[pyspark-ml-feature]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Naveen Nelamali]]></dc:creator>
            <pubDate>Sun, 24 Mar 2024 22:42:13 GMT</pubDate>
            <atom:updated>2024-04-05T13:43:40.701Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Spark Executor Memory Overhead: Understanding & Best Practices]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/sparkbyexamples/spark-executor-memory-overhead-understanding-best-practices-a7e20258474d?source=rss-b76541a95c99------2"><img src="https://cdn-images-1.medium.com/max/1536/1*gxYVBKbg9PyXnJUmlO_14g.png" width="1536"></a></p><p class="medium-feed-snippet">Spark executor memory overhead refers to additional memory allocated beyond the user-defined executor memory in Apache Spark</p><p class="medium-feed-link"><a href="https://medium.com/sparkbyexamples/spark-executor-memory-overhead-understanding-best-practices-a7e20258474d?source=rss-b76541a95c99------2">Continue reading on SparkByExamples »</a></p></div>]]></description>
            <link>https://medium.com/sparkbyexamples/spark-executor-memory-overhead-understanding-best-practices-a7e20258474d?source=rss-b76541a95c99------2</link>
            <guid isPermaLink="false">https://medium.com/p/a7e20258474d</guid>
            <category><![CDATA[data-engineer]]></category>
            <category><![CDATA[apache-spark]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[big-data-analytics]]></category>
            <dc:creator><![CDATA[Naveen Nelamali]]></dc:creator>
            <pubDate>Sun, 14 Jan 2024 04:22:44 GMT</pubDate>
            <atom:updated>2024-01-14T04:47:09.832Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[When to use Hive Partitioning and Bucketing?]]></title>
            <link>https://medium.com/sparkbyexamples/when-to-use-hive-partitioning-and-bucketing-7598785c36fc?source=rss-b76541a95c99------2</link>
            <guid isPermaLink="false">https://medium.com/p/7598785c36fc</guid>
            <category><![CDATA[nosql]]></category>
            <category><![CDATA[apache-hive]]></category>
            <category><![CDATA[big-data-analytics]]></category>
            <category><![CDATA[big-data]]></category>
            <dc:creator><![CDATA[Naveen Nelamali]]></dc:creator>
            <pubDate>Fri, 12 Jan 2024 20:29:04 GMT</pubDate>
            <atom:updated>2024-01-12T20:29:04.614Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RvApsgsNmC_f5tcTXZQaoA.png" /></figure><p>In this article, I will explain Hive Partitioning and Bucketing, the difference between Partitioning vs Bucketing by exploring the advantages and disadvantages of each feature and when to use each with examples.</p><p>At a high level, Hive Partition is a way to split a large table into smaller tables based on the values of a column(one partition for each distinct value) whereas Bucket is a technique to divide the data in a manageable form (you can specify how many buckets you want).</p><p>There are advantages and disadvantages of Partition vs Bucket so you need to choose these based on your data size and the types of Hive queries you run, let’s see the differences in detail.</p><h3>Hive Partitioning vs Bucketing</h3><p>Both Partitioning and Bucketing in Hive are used to improve performance by eliminating table scans when dealing with a large set of data on a Hadoop file system (HDFS). The major difference between Partitioning vs Bucketing lives in the way how they split the data.</p><p>Hive Partition is a way to organize large tables into smaller logical tables based on values of columns; one logical table (partition) for each distinct value. In Hive, tables are created as a directory on HDFS. A table can have one or more partitions that correspond to a sub-directory for each partition inside a table directory.</p><p>let’s assume you have a US census table that contains zip code, city, state, and other columns. Creating a partition on state splits the table into around 50 partitions, when searching for a zipcode within a state (state=’CA’ and zipCode =’92704′) results in faster as it needs to scan only in a state=CA partition directory.</p><p>When creating partitions you have to be very cautious with the number of partitions it creates, as having too many partitions creates too many sub-directories in a table directory which bring unnecessarily and overhead to NameNode since it must keep all metadata for the file system in memory.</p><p>Hive Bucketing a.k.a (Clustering) is a technique to split the data into more manageable files, (By specifying the number of buckets to create). The value of the bucketing column will be hashed by a user-defined number into buckets.</p><p>Bucketing can be created on just one column, you can also create bucketing on a partitioned table to further split the data which further improves the query performance of the partitioned table.</p><p>Each bucket is stored as a file within the table’s directory or the partition&#39;s directories. Note that a partition creates a directory and you can have a partition on one or more columns; these are some of the differences between a Hive partition and a bucket.</p><p>From our example, we already have a partition on the state which leads to around 50 subdirectories on a table directory, and creating a bucketing 10 on zipcode column creates 10 files for each partitioned subdirectory.</p><h3>Some Points to Remember</h3><ul><li>Creating a partition on the zip code of the US population is not a good practice as it creates nearly 42,000 directories on HDFS (US has nearly 42,000 zip codes).</li><li>Since Bucketing works on hashing, if the data is not equally distributed between hashes, it results in in-equal files and may get into performance issues.</li></ul><h3>Hive Partitioning vs Bucketing Example</h3><p>Start your <a href="https://sparkbyexamples.com/apache-hive/connect-to-hive-using-beeline/">Hive Beeline</a> or Hive terminal and run the below statement to understand the syntax.</p><h3>Hive Partitioning Example</h3><p><a href="https://sparkbyexamples.com/apache-hive/hive-create-table-syntax-and-usage-with-examples/">Create the managed table</a> as below with the partition column as a state.</p><pre>CREATE TABLE zipcodes(<br>RecordNumber int,<br>Country string,<br>City string,<br>Zipcode int)<br>PARTITIONED BY(state string)<br>ROW FORMAT DELIMITED<br>FIELDS TERMINATED BY &#39;,&#39;;</pre><p>If you load the zipcodes into this table, you will see the below directories on HDFS.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*xyPCpd6Sp0gGoHjo" /></figure><p>Refer to <a href="https://sparkbyexamples.com/apache-hive/hive-partitions-explained-with-examples/">Hive Partitions with Example</a> to know how to load data into a Partitioned table, show, update, and drop partitions.</p><h3>Hive Bucketing Example</h3><p>In the below example, we are creating a bucketing on zipcode column on top of partitioned by state.</p><pre>CREATE TABLE zipcodes(<br>RecordNumber int,<br>Country string,<br>City string,<br>Zipcode int)<br>PARTITIONED BY(state string)<br>CLUSTERED BY Zipcode INTO 10 BUCKETS<br>ROW FORMAT DELIMITED<br>FIELDS TERMINATED BY &#39;,&#39;;</pre><p>You can also create a bucket on the table column without having partitioned first.</p><h3>Conclusion</h3><p>In this Hive Partitioning vs Bucketing article, you have learned how to improve the performance of the queries by doing Partition and Bucket on Hive tables. These two approaches split the table into defined partitions and/or buckets, which distribute the data into smaller and more manageable parts. This eliminates table scans when you perform queries on partition and bucket columns.</p><p>Happy Learning !!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7598785c36fc" width="1" height="1" alt=""><hr><p><a href="https://medium.com/sparkbyexamples/when-to-use-hive-partitioning-and-bucketing-7598785c36fc">When to use Hive Partitioning and Bucketing?</a> was originally published in <a href="https://medium.com/sparkbyexamples">SparkByExamples</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Do you know you can use Variables in Hive?]]></title>
            <link>https://medium.com/sparkbyexamples/do-you-know-you-can-use-variables-in-hive-17a412146087?source=rss-b76541a95c99------2</link>
            <guid isPermaLink="false">https://medium.com/p/17a412146087</guid>
            <category><![CDATA[apache-hive]]></category>
            <category><![CDATA[nosql]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[big-data-analytics]]></category>
            <dc:creator><![CDATA[Naveen Nelamali]]></dc:creator>
            <pubDate>Fri, 12 Jan 2024 19:38:12 GMT</pubDate>
            <atom:updated>2024-01-12T19:38:12.981Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QZr18SZmWT9ULOg3ktj0oA.png" /></figure><p>Have you every wondered if you can use variables in the Apache Hive database?</p><p>Hive variables are key-value pairs that can be set using the set command and they can be used in scripts and Hive SQL. The values of the variables in Hive scripts are substituted during the query construct.</p><p>In this article, I will explain Hive variables, how to create and set values to the variables and use them on Hive QL and scripts, and finally pass them through the command line.</p><h3>What are the Hive variables?</h3><p>When working with Hive QL and scripts we are often required to use specific values for each environment, and hard-coding these values on code is not a good practice as the values change for each environment.</p><p>To overcome this tight coupling of environment-specific values within the Hive QL script code, we can externalize these by creating variables and setting values outside of the scripts.</p><p>These variables are similar to Unix variables.</p><p>You can set these variables on Hive CLI (older version), Beeline, and Hive scripts.</p><p>Note that when you set values to variables they are local to the active Hive session and these values are not visible to other sessions.</p><h3>Create and Set Hive variables</h3><p>Hive stores variables in four different namespaces, namespace is a way to separate variables.</p><ul><li>hiveconf</li><li>hivevar</li><li>system, and</li><li>env.</li></ul><h4>Hive Config Variables (hiveconf)</h4><p>hiveconf is the default namespace, if you don’t provide a namespace at the time of setting a variable, it will store your variable in hiveconf namespace by default. hiveconf namespace also contains several Hive default configuration variables.</p><p>The below examples set’s emp to table variable in hiveconf namespace.</p><pre>0: jdbc:hive2://127.0.0.1:10000&gt;SET table=&#39;emp&#39;;</pre><p>To retrieve the variable from hiveconf, you have to explicitly specify the namespace. Not specifying the namespace returns an error.</p><pre>0: jdbc:hive2://127.0.0.1:10000&gt;SELECT * FROM ${hiveconf:table}<br># Below statement returns an error<br>0: jdbc:hive2://127.0.0.1:10000&gt;SELECT * FROM ${table}</pre><h4>Hive Custom Variables (hivevar)</h4><p>Hive version 0.8.0 introduced a new namespace hivevar to set the custom variables (JIRA <a href="https://issues.apache.org/jira/browse/HIVE-2020">HIVE-2020</a>), this separates custom variables from Hive default config variables. Hive recommends using hivevar explicitly for custom variables.</p><p>The below example sets the emp value to the table variable in hivevar namespace.</p><pre>0: jdbc:hive2://127.0.0.1:10000&gt;SET hivevar:table=&#39;emp&#39;;</pre><p>To retrieve values from hivevar namespace, you can either specify hivevar namespace or ignore it as hivevar is a default namespace for retrieval.</p><pre>0: jdbc:hive2://127.0.0.1:10000&gt;SELECT * FROM ${table}<br>(or)<br>0: jdbc:hive2://127.0.0.1:10000&gt;SELECT * FROM ${hivevar:table}</pre><h4>Hive System Variables (system)</h4><p>Hive default provides certain system variables and all system variables can be accessed in Hive using system namespace. For example.</p><pre>system:java.version<br>system:user.timezone<br>system:user.home</pre><h4>Hive Environment Variables (env)</h4><p>Hive also default provides certain environment variables and all environment variables can be accessed in Hive using env namespace. For example</p><pre>env:PWD<br>env:USER</pre><h3>3. Set Variables in Hive Scripts</h3><p>Hive scripts support using all variables explained above, you can use any of these along with their namespace.</p><p>let’s create test.hql file with the below content. If you notice, I am referring to the table name from hivevar namespace.</p><pre>0: jdbc:hive2://127.0.0.1:10000&gt;SELECT * FROM ${hivevar:table};</pre><p>Execute the test.hql script by running the below command.</p><pre>bin/beeline -u jdbc:hive2://127.0.0.1:10000 scott tiger -f test.hql<br>(or)<br>bin/hive -f test.hql</pre><p>Before executing the queries mentioned in the script, hive replaces the variables with the actual values from the namespace during the query construct stage.</p><h3>4. How to set and pass command line values to Hive scripts?</h3><p>Hive also supports setting a variable from the command line when <a href="https://sparkbyexamples.com/apache-hive/connect-to-hive-using-beeline/">starting a Hive CLI or beeline</a>.</p><pre>bin/beeline -u jdbc:hive2://127.0.0.1:10000 scott tiger --hivevar table=&#39;emp&#39;<br>bin/hive --hivevar table=&#39;emp&#39;</pre><p>You can refer to this variable in Hive CLI.</p><pre>0: jdbc:hive2://127.0.0.1:10000&gt;SELECT &amp; FROM ${hivevar:table};</pre><p>You can also call test.hql script by setting command line variables.</p><pre>bin/beeline -u jdbc:hive2://127.0.0.1:10000 scott tiger --hivevar table=&#39;dept&#39; -f test.hql<br>(or)<br>bin/hive --hivevar table=&#39;dept&#39; -f test.hql</pre><p>This should select data from the dept table.</p><h3>5. List all Hive variables</h3><p>From Hive scripts you can access the environment (env), system, hive configuration, and custom variables. since there are many variables it’s hard to remember all of them, use a set to list variables in the hive terminal</p><pre>0: jdbc:hive2://127.0.0.1:10000&gt;set;<br>(or)<br>0: jdbc:hive2://127.0.0.1:10000&gt;set -v; #This lists all variables with namespace</pre><p>This set command lists all available variables and configurations in Hive.</p><h3>6. How variable substitute works?</h3><p>Hive substitutes the value for a variable when a query is constructed with the variable.</p><ul><li>Values on Hive variables are visible only to active sessions where it’s been assigned and they cannot be accessible from another session.</li></ul><h3>7. Disabling Variable Substitution</h3><p>By default Hive substitutes all variables, you can disable these using (<a href="https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-hive.variable.substitute">hive.variable.substitute</a>=true) in case you want to run a script without substitution variables.</p><pre>0: jdbc:hive2://127.0.0.1:10000&gt;set hive.variable.substitute=false;</pre><p>I hope this helps!!</p><p>Happy Learning !!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=17a412146087" width="1" height="1" alt=""><hr><p><a href="https://medium.com/sparkbyexamples/do-you-know-you-can-use-variables-in-hive-17a412146087">Do you know you can use Variables in Hive?</a> was originally published in <a href="https://medium.com/sparkbyexamples">SparkByExamples</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Union Pandas DataFrames?]]></title>
            <link>https://medium.com/sparkbyexamples/how-to-union-pandas-dataframes-a9b0736495d9?source=rss-b76541a95c99------2</link>
            <guid isPermaLink="false">https://medium.com/p/a9b0736495d9</guid>
            <category><![CDATA[data-engineer]]></category>
            <category><![CDATA[pandas-dataframe]]></category>
            <category><![CDATA[big-data-analytics]]></category>
            <category><![CDATA[python-programming]]></category>
            <dc:creator><![CDATA[Naveen Nelamali]]></dc:creator>
            <pubDate>Fri, 12 Jan 2024 05:52:37 GMT</pubDate>
            <atom:updated>2024-01-12T05:54:34.793Z</atom:updated>
            <content:encoded><![CDATA[<h3>How to Union Pandas DataFrames by Vertical &amp; Horizontal</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ka8c1HXIMTMd1G6wsMWF4g.png" /></figure><p>In pandas, you can use the concat() function to concatenate or union the DataFrames along with a particular axis (either rows or columns). To union the Pandas DataFrames use the concat() function, by either vertical concatenation (concatenating along rows) or horizontal concatenation (concatenating along columns).</p><p>In this article, I will explain how to union two pandas DataFrames by rows and columns with examples.</p><h3>Create Two DataFrames</h3><p>Let’s create two DataFrames with a few rows and columns.</p><pre># Create DataFrame<br>import pandas as pd<br>technologies = [ [&quot;Spark&quot;,20000,&quot;30days&quot;], <br>                 [&quot;Pandas&quot;,25000,&quot;40days&quot;], <br>               ]<br>column_names=[&quot;Courses&quot;,&quot;Fee&quot;,&#39;Duration&#39;]<br>df=pd.DataFrame(technologies, columns=column_names)<br>print(&quot;First DataFrame:\n&quot;,df)<br><br># Create second DataFrame<br>technologies = [ [&quot;Hadoop&quot;,25000,&quot;50days&quot;], <br>                 [&quot;Java&quot;,30000,&quot;40days&quot;], <br>               ]<br>column_names=[&quot;Courses&quot;,&quot;Fee&quot;,&#39;Duration&#39;]<br>df1=pd.DataFrame(technologies, columns=column_names)<br>print(&quot;Second DataFrame:\n&quot;,df1)</pre><p>Yields below output</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/260/0*z9tLb95kQwEIBomq" /></figure><h3>Vertical Union of Pandas DataFrames (Concatenation along Rows)</h3><p>To concatenate (union) pandas DataFrames using the concat() method to concat two DataFrames by rows meaning appending two DataFrames. By default, it performs append operations similar to a union where it bright all rows from both DataFrames to a single DataFrame.</p><p>In the below example, concatenates the DataFrames df and df1 along the rows (axis=0), performing a union operation. The resulting DataFrame (df2) contains all the rows from both original DataFrames, and the index is not reset, meaning it retains the original indices.</p><pre># Union pandas dataframes using concat<br>df2 = pd.concat([df, df1])<br>print(&quot;Union pandas dataframes using concat:\n&quot;, df2)</pre><p>Yields below output.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/305/0*4poS66cmA3OFDvl_" /></figure><h3>Concatenating Along Rows (axis=0)</h3><p>If you want to perform a union operation on two Pandas DataFrames using concat(), you can concatenate them along the rows (axis=0). For instance, ignore_index=True is used to reset the index of the resulting DataFrame.</p><pre># Concatenate along rows (performing a union)<br>df2 = pd.concat([df, df1], ignore_index=True)<br>print(&quot;Union pandas DataFrames and reset index:\n&quot;, df2)<br><br># Output:<br>#Union pandas DataFrames and reset index:<br>#   Courses    Fee Duration<br>#0   Spark  20000   30days<br>#1  Pandas  25000   40days<br>#2  Hadoop  25000   50days<br>#3    Java  30000   40days</pre><p>As you can see, the resulting DataFrame includes all unique rows from both original DataFrames, performing a union operation. If there are duplicate rows, they will be retained in the result.</p><h3>5. Horizontal Union (Concatenation along Columns)</h3><p>Alternatively, if you want to concatenate DataFrames along columns, you can use axis=1 as an argument in the pd.concat() function.</p><pre># Concatenate pandas DataFrames along columns<br>df2 = pd.concat([df, df1], axis=1)<br>print(&quot;Concatenate pandas DataFrames along columns:\n&quot;, df2)<br><br># Output<br>#Concatenate pandas DataFrames along columns:<br>#   Courses    Fee Duration Courses    Fee Duration<br>#0   Spark  20000   30days  Hadoop  25000   50days<br>#1  Pandas  25000   40days    Java  30000   40days</pre><h3>6. Concatenate with keys</h3><p>When you use the keys parameter with pd.concat(), it allows you to create a hierarchical index based on the provided keys.</p><p>In the below example, the resulting DataFrame has a multi-level index where the first level corresponds to the keys (‘df1’ and ‘df2’), and the second level corresponds to the original index of the DataFrames.</p><pre># Concatenate with keys<br>result = pd.concat([df, df1], keys=[&#39;df&#39;, &#39;df1&#39;])<br>print(&quot;Concatenate with keys:\n&quot;,result)<br><br># Union with keys<br>result = pd.concat([df, df1], keys=[&#39;df&#39;, &#39;df1&#39;]).drop_duplicates()<br>print(&quot;Union with keys:\n&quot;,result)<br><br># Output:<br># Concatenate with keys:<br>#       Courses    Fee Duration<br># df  0   Spark  20000   30days<br>#     1  Pandas  25000   40days<br># df1 0  Hadoop  25000   50days<br>#     1    Java  30000   40days</pre><p>Hope you like it.</p><p>Happy Learning !!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a9b0736495d9" width="1" height="1" alt=""><hr><p><a href="https://medium.com/sparkbyexamples/how-to-union-pandas-dataframes-a9b0736495d9">How to Union Pandas DataFrames?</a> was originally published in <a href="https://medium.com/sparkbyexamples">SparkByExamples</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why Avoid UDFs in Spark & PySpark?]]></title>
            <link>https://medium.com/sparkbyexamples/why-avoid-udfs-in-spark-pyspark-e5903050e33c?source=rss-b76541a95c99------2</link>
            <guid isPermaLink="false">https://medium.com/p/e5903050e33c</guid>
            <category><![CDATA[apache-spark]]></category>
            <category><![CDATA[spark-udf]]></category>
            <category><![CDATA[pyspark-udf]]></category>
            <dc:creator><![CDATA[Naveen Nelamali]]></dc:creator>
            <pubDate>Fri, 12 Jan 2024 04:53:55 GMT</pubDate>
            <atom:updated>2024-01-12T05:06:17.595Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EmphAOLyQ0YWbt9Wnrx0IQ.jpeg" /></figure><p>User-Defined Functions (UDFs) are a powerful feature in Apache Spark and PySpark that allow users to define their own custom functions to perform complex data operations. However, while UDFs can be useful in certain situations, they can also be a source of performance issues and other problems in Spark applications. In this article, we will explore what makes UDFs problematic in Spark and PySpark, and what alternatives exist for performing data operations that can help overcome these issues.</p><h3>What is Spark UDF and Why do we need it?</h3><p>UDFs are custom functions written by users that can be applied to data in a Spark/PySpark DataFrame or RDD. They can be helpful when working with complex data types or performing non-standard operations that are not supported by Spark’s built-in functions.</p><p>Below is an example of UDF in Spark with Scala</p><pre>import org.apache.spark.sql.functions.udf<br>import org.apache.spark.sql.functions.col<br>import org.apache.spark.sql.{Row, SparkSession}<br><br>object SparkUDF extends App{<br><br>  //Create spark session object<br><br>  import spark.implicits._<br>  val columns = Seq(&quot;Seqno&quot;,&quot;Quote&quot;)<br>  val data = Seq((&quot;1&quot;, &quot;Be the change that you wish to see in the world&quot;))<br>  val df = data.toDF(columns:_*)<br><br>  // Create UDF<br>  val convertCase =  (str:String) =&gt; {<br>    val arr = str.split(&quot; &quot;)<br>    arr.map(f=&gt;  f.substring(0,1).toUpperCase + f.substring(1,f.length)).mkString(&quot; &quot;)<br>  }<br>  val convertUDF = udf(convertCase)<br><br>  // Use UDF on DataFrame<br>  df.select(col(&quot;Seqno&quot;),<br>    convertUDF(col(&quot;Quote&quot;)).as(&quot;Quote&quot;) ).show(false)<br>}</pre><h3>Why using UDFs are Bottle-neck to Performance</h3><p>User-Defined Functions (UDFs) in Spark can incur performance issues due to serialization overhead, necessitating the conversion of data between internal and external representations. Additionally, UDFs may trigger data movement across nodes, introducing network overhead.</p><p>Spark struggles to optimize complex UDF logic, leading to suboptimal execution plans and reduced performance. Manual resource management for UDFs can be challenging, impacting overall job efficiency.</p><h3>Why to Avoid UDFs in PySpark</h3><p>One of the main issues with UDFs is that they can be slow to execute. Since UDFs are written in Python, they are not compiled like PySpark’s built-in functions, which are implemented in Java. This means that UDFs can be significantly slower when processing large datasets, especially when compared to native PySpark functions. Additionally, UDFs can be memory-intensive, which can lead to out-of-memory errors when working with large datasets.</p><h3>Disadvantages of UDF</h3><p>While User-Defined Functions (UDFs) in Spark offer flexibility and extensibility, there are some disadvantages associated with their usage. Here are some drawbacks of using UDFs in Spark:</p><p><strong>Performance Overhead:</strong></p><ul><li>UDFs introduce some performance overhead compared to built-in Spark functions. The execution of UDFs involves the cost of serialization, data transfer, and deserialization.</li></ul><p><strong>Limited Optimization Opportunities:</strong></p><ul><li>Spark might not be able to optimize UDFs as effectively as built-in functions. This could lead to suboptimal execution plans and, consequently, slower performance.</li></ul><p><strong>Type Safety and Error Handling:</strong></p><ul><li>UDFs might not provide the same level of type safety and error handling as built-in functions. Incorrect data types or errors within UDFs can lead to runtime issues that are harder to debug.</li></ul><p><strong>Data Movement Overhead:</strong></p><ul><li>When using UDFs, there could be additional data movement between nodes in a distributed environment, leading to increased network overhead.</li></ul><p><strong>Resource Management:</strong></p><ul><li>Managing resources (memory, CPU) for UDFs might require manual tuning, and improper resource allocation could impact overall job performance.</li></ul><p><strong>Limited Language Support:</strong></p><ul><li>UDFs are typically written in languages like Scala, Java, or Python. While this covers a broad range of use cases, it might not be as versatile as using native Spark functions for certain operations.</li></ul><p><strong>Dependency Management:</strong></p><ul><li>UDFs might have external dependencies that need to be managed separately. Ensuring that these dependencies are available across all worker nodes can be challenging.</li></ul><p><strong>Debugging Complexity:</strong></p><ul><li>Debugging UDFs can be more challenging than debugging Spark SQL or DataFrame operations. The development and debugging process might be less user-friendly.</li></ul><p><strong>Compatibility with Spark Ecosystem:</strong></p><ul><li>Some components of the Spark ecosystem, such as Spark Streaming or Spark MLlib, might not seamlessly support UDFs, limiting their use in certain contexts.</li></ul><p><strong>Potential for Non-Optimized Execution Plans:</strong></p><ul><li>Depending on the complexity of the UDF logic, Spark might struggle to generate optimized execution plans, leading to inefficient query execution.</li></ul><p>Despite these disadvantages, UDFs remain a valuable tool for extending Spark’s functionality when necessary. Careful consideration of when and how to use UDFs is essential to mitigate these drawbacks and ensure optimal performance.</p><h3>Conclusion</h3><p>In conclusion, UDFs can be helpful when working with complex data types or performing non-standard operations in PySpark. However, they come with a number of downsides that can make them problematic in larger-scale applications.</p><p>To overcome these issues, it is important to consider using PySpark’s built-in functions, SQL functions, and libraries whenever possible. These alternatives are optimized for performance and are easier to distribute and debug than UDFs, making them a more practical choice for large-scale data processing applications.</p><p>Happy Learning !!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e5903050e33c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/sparkbyexamples/why-avoid-udfs-in-spark-pyspark-e5903050e33c">Why Avoid UDFs in Spark &amp; PySpark?</a> was originally published in <a href="https://medium.com/sparkbyexamples">SparkByExamples</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What are the Specific Roles of Spark Driver and Executor]]></title>
            <link>https://medium.com/sparkbyexamples/what-are-the-specific-roles-of-spark-driver-and-executor-a17f90c7e7fd?source=rss-b76541a95c99------2</link>
            <guid isPermaLink="false">https://medium.com/p/a17f90c7e7fd</guid>
            <category><![CDATA[pyspark]]></category>
            <category><![CDATA[apache-spark-executor]]></category>
            <category><![CDATA[apache-spark]]></category>
            <category><![CDATA[apache-spark-driver]]></category>
            <dc:creator><![CDATA[Naveen Nelamali]]></dc:creator>
            <pubDate>Fri, 12 Jan 2024 04:26:31 GMT</pubDate>
            <atom:updated>2024-01-12T19:24:05.216Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nJum-35aJ-9jEVR8bWPKMA.jpeg" /></figure><p>Have you ever wondered what the different roles of Apache Spark Driver and Executor play when running your application in a distributed cluster?</p><p>Apache Spark is a popular open-source distributed computing system that is widely used for big data processing and analytics. When running Spark applications, two crucial components of the Spark architecture are the Spark Driver and Executor.</p><p>While both are responsible for managing tasks and resources, they have distinct roles and responsibilities in the overall execution of a Spark job. In this article, we will explore the specific roles of the Spark Driver and Executor, and how they work together to execute Spark applications efficiently and effectively.</p><h3>What is the role of Spark Driver?</h3><p>The Apache Spark Driver is the program that declares the SparkContext, which is responsible for converting the user program into a series of tasks that can be distributed across the cluster. It also coordinates the execution of tasks and communicates with the Cluster Manager to allocate resources for the application. In short, the Spark Driver plays a crucial role in managing the overall execution of a Spark application.</p><p><strong>Spark Driver Configurations</strong></p><p>You need to specify the driver configurations when submitting the spark application using spark-submit command. Following are some of them you can use based on your needs. For a complete list, refer to the Spark official documentation.</p><ul><li>--driver-memory: Specifies the amount of memory to allocate for the driver (e.g., &quot;1g&quot; for 1 gigabyte).</li><li>--driver-cores: Specifies the number of cores to allocate for the driver.</li><li>--driver-class-path: Specifies the classpath for the driver.</li><li>--driver-java-options: Specifies extra Java options for the driver.</li><li>--driver-library-path: Specifies the library path for the driver.</li><li>--conf spark.driver.maxResultSize: Limits the maximum size of results that Spark will return to the driver.</li><li>--conf spark.driver.host: Specifies the hostname or IP address the driver runs on.</li></ul><h3>What is the role of Spark Executor?</h3><p>In Apache Spark, an Executor is a process that is responsible for running the tasks in parallel on worker nodes in the cluster. The Spark driver program launches the Executor process, and it runs on the worker nodes to execute the tasks the driver assigns. Hence, it is critical to understand the difference between Spark Driver and Executor.</p><p>The Executor is a key component of the Spark runtime architecture because it is responsible for processing the data and executing the code in parallel on the worker nodes. The Executor runs the user-defined Spark code, which can be written in various programming languages such as Scala, Java, Python or R, and it performs the necessary calculations and transformations on the data using the RDD (Resilient Distributed Dataset) API or the DataFrame API.</p><p><strong>Spark Executor Configurations</strong></p><ul><li>--executor-memory: Specifies the amount of memory to allocate per executor (e.g., &quot;1g&quot; for 1 gigabyte).</li><li>--num-executors: Specifies the number of executors to launch.</li><li>--executor-cores: Specifies the number of cores to allocate for each executor.</li><li>--conf spark.executor.extraClassPath: Specifies extra classpath entries for executors.</li><li>--conf spark.executor.extraJavaOptions: Specifies extra Java options for executors.</li><li>--conf spark.executor.extraLibraryPath: Specifies extra library path entries for executors.</li><li>--conf spark.yarn.executor.memoryOverhead: Specifies the amount of non-heap memory to be allocated per executor.</li><li>--conf spark.executor.memoryFraction: Specifies the fraction of the heap space that is allocated for Spark&#39;s memory management.</li><li>--conf spark.dynamicAllocation.enabled: Enables or disables dynamic allocation of executors.</li></ul><p><strong>Spark Executor is a fault-tolerant</strong></p><p>The Executor is designed to be fault-tolerant and resilient to failures, which is essential for handling large-scale data processing workloads. If an executor fails due to hardware or software issues, the Spark framework automatically re-launches the Executor on a different worker node and re-runs the failed tasks to ensure that the processing is not interrupted.</p><p>The number of Executors and their configuration parameters such as memory, cores, and parallelism can be adjusted based on the specific requirements of the Spark application and the resources available in the cluster. The optimal configuration can help to improve the performance and scalability of the Spark application and allow it to process large amounts of data efficiently.</p><h3>Conclusion</h3><p>In summary, the Spark Driver is the central control and coordination point for a Spark application, managing the overall execution, while Spark Executors are responsible for executing specific tasks on worker nodes, each managing its own memory space.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a17f90c7e7fd" width="1" height="1" alt=""><hr><p><a href="https://medium.com/sparkbyexamples/what-are-the-specific-roles-of-spark-driver-and-executor-a17f90c7e7fd">What are the Specific Roles of Spark Driver and Executor</a> was originally published in <a href="https://medium.com/sparkbyexamples">SparkByExamples</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Introduction to Spark-Submit: A Comprehensive Guide to Submitting Spark Applications]]></title>
            <link>https://medium.com/sparkbyexamples/introduction-to-spark-submit-a-comprehensive-guide-to-submitting-spark-applications-f6ca2314766c?source=rss-b76541a95c99------2</link>
            <guid isPermaLink="false">https://medium.com/p/f6ca2314766c</guid>
            <category><![CDATA[submit-pyspark-job]]></category>
            <category><![CDATA[submit-spark-application]]></category>
            <category><![CDATA[spark-submit-command]]></category>
            <category><![CDATA[apache-spark]]></category>
            <dc:creator><![CDATA[Naveen Nelamali]]></dc:creator>
            <pubDate>Thu, 11 Jan 2024 21:36:57 GMT</pubDate>
            <atom:updated>2024-01-11T22:21:41.223Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PFBWwtpe5jQt7ASUVmklvg.jpeg" /></figure><h3>What is Spark Sumit?</h3><p>Spark Submit is a command-line tool that comes with Apache Spark, a powerful open-source distributed computing system designed for large-scale data processing.</p><h3>Why do you need spark-submit Command?</h3><p>Spark Submit allows users to submit their Spark applications to a cluster for execution. It is a crucial component of the Spark ecosystem and plays a vital role in developing, testing, and deploying Spark applications.</p><p>Apache Spark applications are written in various programming languages such as Scala, Java, Python, and R. Once the application is developed, it needs to be submitted to a cluster for execution. This is where Spark Submit comes into play. By using Spark Submit, users can submit their applications to the cluster in a few simple steps.</p><h3>Spark Submit Configurations</h3><p>One of the significant advantages of using Spark Submit is that it allows users to specify various configuration parameters while submitting the applications. These configuration parameters include cluster URL, application name, number of executors, executor memory, and many others.</p><h4>Configure Driver Memory</h4><p>Spark Driver Memory refers to the amount of memory allocated to the driver process. The driver is the process responsible for coordinating the tasks and executing the application on the cluster.</p><p>The amount of memory allocated to the driver is important because it determines the maximum amount of data that can be processed by the driver. If the driver memory is too low, it may cause out-of-memory errors or slow down the application. On the other hand, if the driver memory is too high, it may lead to unnecessary memory allocation and overhead, which can reduce the performance of the application.</p><p>The default value of driver memory in Spark is 1g, but it can be increased or decreased using the ` — driver-memory` command-line option or the `spark.driver.memory` configuration property.</p><h4>Configure Executor Memory</h4><p>The amount of memory allocated to the executor is important because it determines the maximum amount of data that can be processed by each task. If the executor memory is too low, it may cause out-of-memory errors or slow down the application. On the other hand, if the executor memory is too high, it may lead to unnecessary memory allocation and overhead, which can reduce the performance of the application.</p><p>The default value of executor memory in Spark is 1g, but it can be increased or decreased using the ` — executor-memory` command-line option or the `spark.executor.memory` configuration property.</p><h4>Configure Executor Core</h4><p>Executor Cores refers to the number of parallel processing units available for a single executor process on a worker node in the cluster. Each executor can have one or more cores assigned to it, depending on the available resources and the requirements of the Spark application.</p><p>The optimal number of executor cores can be determined based on the specific requirements of the application, the available resources in the cluster, and the size of the data being processed. As a general rule, it is recommended to allocate 5 cores per executor for CPU-bound tasks and 1 core per executor for I/O-bound tasks.</p><h3>What are the modes of Submitting Spark Applications?</h3><p>There are two modes in which Spark Submit can be run Client mode and Cluster mode. Use --deploy-mode to specify the mode.</p><p><strong>Client mode:</strong> The Spark driver runs on the client machine, and the application is submitted to the cluster for execution.</p><p><strong>Cluster mode:</strong> The Spark driver runs on one of the nodes in the cluster, and the application is also executed on the cluster.</p><h3>What Cluster Managers Does Spark Support?</h3><p>Apache Spark supports submitting applications to different cluster managers. Use --master to specify the master</p><p>Yarn, Mesos, Standalone. Kubernetes and local</p><p><strong>Yarn:</strong> Use yarn if your cluster resources are managed by Hadoop Yarn.</p><p><strong>Mesos:</strong> Use mesos://HOST:PORT for Mesos cluster manager, replace the host and port of Mesos cluster manager.</p><p><strong>Standalone: </strong>Use spark://HOST:PORT for the Standalone cluster, replace the host and port of stand-alone cluster.</p><p><strong>Kubernetes: </strong>Use k8s://HOST:PORT for Kubernetes, replace the host and port of Kubernetes. This by default connects with https, but if you want to use unsecured use k8s://https://HOST:PORT</p><p><strong>local: </strong>Use local to run locally with a one-worker thread.<br>Use local[k] and specify k with the number of cores you have locally, this runs the application with k worker threads.<br>use local[k,F] and specify F with number of attempts it should run when failed.</p><h3>How do you Run Spark Application by the spark-submit command?</h3><p>The first step in submitting a Spark application is to create a Spark application package. A Spark application package is a bundle of all the files required to run the application. This includes the code files, configuration files, and any dependent libraries. Once the application package is created, it needs to be submitted to the cluster using Spark Submit.</p><pre>./bin/spark-submit \<br>  --master &lt;master-url&gt; \<br>  --deploy-mode &lt;deploy-mode&gt; \<br>  --conf &lt;key&lt;=&lt;value&gt; \<br>  --driver-memory &lt;value&gt;g \<br>  --executor-memory &lt;value&gt;g \<br>  --executor-cores &lt;number of cores&gt;  \<br>  --jars  &lt;comma separated dependencies&gt;<br>  --class &lt;main-class&gt; \<br>  &lt;application-jar&gt; \<br>  [application-arguments]</pre><p>Here is an example of submitting a spark application</p><pre>./bin/spark2-submit \<br>   --master yarn \<br>   --deploy-mode cluster \<br>   --driver-memory 8g \<br>   --executor-memory 16g \<br>   --executor-cores 2  \<br>   --files /path/log4j.properties,/path/file2.conf,/path/file3.json \<br>   --class org.apache.spark.examples.SparkPi \<br>   /spark-home/examples/jars/spark-examples_replace-spark-version.jar 80</pre><p>These parameters can significantly impact the performance and resource utilization of the application. By being able to specify these parameters, users can optimize their application’s performance and resource utilization.</p><h3>How to Submit PySpark Application?</h3><p>Similarly, you can also submit a PySpark (Spark with Python) application using the same spark-submit command.</p><pre>./bin/spark-submit \<br>   --master yarn \<br>   --deploy-mode cluster \<br>   --driver-memory 8g \<br>   --executor-memory 16g \<br>   --executor-cores 2  \<br>   --py-files file1.py,file2.py,file3.zip<br>   wordByExample.py</pre><p>Another significant advantage of using Spark Submit is that it handles the complexities of the distributed environment. When an application is submitted to a cluster, Spark Submit takes care of distributing the application files, setting up the environment, launching the driver program, and managing the execution of the application. This makes it much easier for users to deploy and manage their Spark applications.</p><h3>Conclusion</h3><p>In conclusion, Spark Submit is a command-line tool that is an integral part of the Spark ecosystem. It allows users to submit Spark applications to a cluster for execution and provides various functionalities such as running examples, testing applications, and performing administrative tasks. By being able to specify configuration parameters and handling the complexities of the distributed environment, Spark Submit makes it easier for users to develop, test, and deploy their Spark applications.</p><p>Happy Learning !!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f6ca2314766c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/sparkbyexamples/introduction-to-spark-submit-a-comprehensive-guide-to-submitting-spark-applications-f6ca2314766c">Introduction to Spark-Submit: A Comprehensive Guide to Submitting Spark Applications</a> was originally published in <a href="https://medium.com/sparkbyexamples">SparkByExamples</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>