<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[SparkByExamples - Medium]]></title>
        <description><![CDATA[SparkByExamples is a BigData, Data Engineering, Machine Learning, AI, and Cloud platform knowledge-sharing publication platform with the intent to share the knowledge that we come across in real-time projects. - Medium]]></description>
        <link>https://medium.com/sparkbyexamples?source=rss----fe7db074b243---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>SparkByExamples - Medium</title>
            <link>https://medium.com/sparkbyexamples?source=rss----fe7db074b243---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 08:31:13 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/sparkbyexamples" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Spark Executor Memory Overhead: Understanding & Best Practices]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/sparkbyexamples/spark-executor-memory-overhead-understanding-best-practices-a7e20258474d?source=rss----fe7db074b243---4"><img src="https://cdn-images-1.medium.com/max/1536/1*gxYVBKbg9PyXnJUmlO_14g.png" width="1536"></a></p><p class="medium-feed-snippet">Spark executor memory overhead refers to additional memory allocated beyond the user-defined executor memory in Apache Spark</p><p class="medium-feed-link"><a href="https://medium.com/sparkbyexamples/spark-executor-memory-overhead-understanding-best-practices-a7e20258474d?source=rss----fe7db074b243---4">Continue reading on SparkByExamples »</a></p></div>]]></description>
            <link>https://medium.com/sparkbyexamples/spark-executor-memory-overhead-understanding-best-practices-a7e20258474d?source=rss----fe7db074b243---4</link>
            <guid isPermaLink="false">https://medium.com/p/a7e20258474d</guid>
            <category><![CDATA[data-engineer]]></category>
            <category><![CDATA[apache-spark]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[big-data-analytics]]></category>
            <dc:creator><![CDATA[Naveen Nelamali]]></dc:creator>
            <pubDate>Sun, 14 Jan 2024 04:23:28 GMT</pubDate>
            <atom:updated>2024-01-14T04:47:09.832Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[When to use Hive Partitioning and Bucketing?]]></title>
            <link>https://medium.com/sparkbyexamples/when-to-use-hive-partitioning-and-bucketing-7598785c36fc?source=rss----fe7db074b243---4</link>
            <guid isPermaLink="false">https://medium.com/p/7598785c36fc</guid>
            <category><![CDATA[nosql]]></category>
            <category><![CDATA[apache-hive]]></category>
            <category><![CDATA[big-data-analytics]]></category>
            <category><![CDATA[big-data]]></category>
            <dc:creator><![CDATA[Naveen Nelamali]]></dc:creator>
            <pubDate>Fri, 12 Jan 2024 20:29:55 GMT</pubDate>
            <atom:updated>2024-01-12T20:29:04.614Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RvApsgsNmC_f5tcTXZQaoA.png" /></figure><p>In this article, I will explain Hive Partitioning and Bucketing, the difference between Partitioning vs Bucketing by exploring the advantages and disadvantages of each feature and when to use each with examples.</p><p>At a high level, Hive Partition is a way to split a large table into smaller tables based on the values of a column(one partition for each distinct value) whereas Bucket is a technique to divide the data in a manageable form (you can specify how many buckets you want).</p><p>There are advantages and disadvantages of Partition vs Bucket so you need to choose these based on your data size and the types of Hive queries you run, let’s see the differences in detail.</p><h3>Hive Partitioning vs Bucketing</h3><p>Both Partitioning and Bucketing in Hive are used to improve performance by eliminating table scans when dealing with a large set of data on a Hadoop file system (HDFS). The major difference between Partitioning vs Bucketing lives in the way how they split the data.</p><p>Hive Partition is a way to organize large tables into smaller logical tables based on values of columns; one logical table (partition) for each distinct value. In Hive, tables are created as a directory on HDFS. A table can have one or more partitions that correspond to a sub-directory for each partition inside a table directory.</p><p>let’s assume you have a US census table that contains zip code, city, state, and other columns. Creating a partition on state splits the table into around 50 partitions, when searching for a zipcode within a state (state=’CA’ and zipCode =’92704′) results in faster as it needs to scan only in a state=CA partition directory.</p><p>When creating partitions you have to be very cautious with the number of partitions it creates, as having too many partitions creates too many sub-directories in a table directory which bring unnecessarily and overhead to NameNode since it must keep all metadata for the file system in memory.</p><p>Hive Bucketing a.k.a (Clustering) is a technique to split the data into more manageable files, (By specifying the number of buckets to create). The value of the bucketing column will be hashed by a user-defined number into buckets.</p><p>Bucketing can be created on just one column, you can also create bucketing on a partitioned table to further split the data which further improves the query performance of the partitioned table.</p><p>Each bucket is stored as a file within the table’s directory or the partition&#39;s directories. Note that a partition creates a directory and you can have a partition on one or more columns; these are some of the differences between a Hive partition and a bucket.</p><p>From our example, we already have a partition on the state which leads to around 50 subdirectories on a table directory, and creating a bucketing 10 on zipcode column creates 10 files for each partitioned subdirectory.</p><h3>Some Points to Remember</h3><ul><li>Creating a partition on the zip code of the US population is not a good practice as it creates nearly 42,000 directories on HDFS (US has nearly 42,000 zip codes).</li><li>Since Bucketing works on hashing, if the data is not equally distributed between hashes, it results in in-equal files and may get into performance issues.</li></ul><h3>Hive Partitioning vs Bucketing Example</h3><p>Start your <a href="https://sparkbyexamples.com/apache-hive/connect-to-hive-using-beeline/">Hive Beeline</a> or Hive terminal and run the below statement to understand the syntax.</p><h3>Hive Partitioning Example</h3><p><a href="https://sparkbyexamples.com/apache-hive/hive-create-table-syntax-and-usage-with-examples/">Create the managed table</a> as below with the partition column as a state.</p><pre>CREATE TABLE zipcodes(<br>RecordNumber int,<br>Country string,<br>City string,<br>Zipcode int)<br>PARTITIONED BY(state string)<br>ROW FORMAT DELIMITED<br>FIELDS TERMINATED BY &#39;,&#39;;</pre><p>If you load the zipcodes into this table, you will see the below directories on HDFS.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*xyPCpd6Sp0gGoHjo" /></figure><p>Refer to <a href="https://sparkbyexamples.com/apache-hive/hive-partitions-explained-with-examples/">Hive Partitions with Example</a> to know how to load data into a Partitioned table, show, update, and drop partitions.</p><h3>Hive Bucketing Example</h3><p>In the below example, we are creating a bucketing on zipcode column on top of partitioned by state.</p><pre>CREATE TABLE zipcodes(<br>RecordNumber int,<br>Country string,<br>City string,<br>Zipcode int)<br>PARTITIONED BY(state string)<br>CLUSTERED BY Zipcode INTO 10 BUCKETS<br>ROW FORMAT DELIMITED<br>FIELDS TERMINATED BY &#39;,&#39;;</pre><p>You can also create a bucket on the table column without having partitioned first.</p><h3>Conclusion</h3><p>In this Hive Partitioning vs Bucketing article, you have learned how to improve the performance of the queries by doing Partition and Bucket on Hive tables. These two approaches split the table into defined partitions and/or buckets, which distribute the data into smaller and more manageable parts. This eliminates table scans when you perform queries on partition and bucket columns.</p><p>Happy Learning !!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7598785c36fc" width="1" height="1" alt=""><hr><p><a href="https://medium.com/sparkbyexamples/when-to-use-hive-partitioning-and-bucketing-7598785c36fc">When to use Hive Partitioning and Bucketing?</a> was originally published in <a href="https://medium.com/sparkbyexamples">SparkByExamples</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Do you know you can use Variables in Hive?]]></title>
            <link>https://medium.com/sparkbyexamples/do-you-know-you-can-use-variables-in-hive-17a412146087?source=rss----fe7db074b243---4</link>
            <guid isPermaLink="false">https://medium.com/p/17a412146087</guid>
            <category><![CDATA[apache-hive]]></category>
            <category><![CDATA[nosql]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[big-data-analytics]]></category>
            <dc:creator><![CDATA[Naveen Nelamali]]></dc:creator>
            <pubDate>Fri, 12 Jan 2024 19:38:24 GMT</pubDate>
            <atom:updated>2024-01-12T19:38:12.981Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QZr18SZmWT9ULOg3ktj0oA.png" /></figure><p>Have you every wondered if you can use variables in the Apache Hive database?</p><p>Hive variables are key-value pairs that can be set using the set command and they can be used in scripts and Hive SQL. The values of the variables in Hive scripts are substituted during the query construct.</p><p>In this article, I will explain Hive variables, how to create and set values to the variables and use them on Hive QL and scripts, and finally pass them through the command line.</p><h3>What are the Hive variables?</h3><p>When working with Hive QL and scripts we are often required to use specific values for each environment, and hard-coding these values on code is not a good practice as the values change for each environment.</p><p>To overcome this tight coupling of environment-specific values within the Hive QL script code, we can externalize these by creating variables and setting values outside of the scripts.</p><p>These variables are similar to Unix variables.</p><p>You can set these variables on Hive CLI (older version), Beeline, and Hive scripts.</p><p>Note that when you set values to variables they are local to the active Hive session and these values are not visible to other sessions.</p><h3>Create and Set Hive variables</h3><p>Hive stores variables in four different namespaces, namespace is a way to separate variables.</p><ul><li>hiveconf</li><li>hivevar</li><li>system, and</li><li>env.</li></ul><h4>Hive Config Variables (hiveconf)</h4><p>hiveconf is the default namespace, if you don’t provide a namespace at the time of setting a variable, it will store your variable in hiveconf namespace by default. hiveconf namespace also contains several Hive default configuration variables.</p><p>The below examples set’s emp to table variable in hiveconf namespace.</p><pre>0: jdbc:hive2://127.0.0.1:10000&gt;SET table=&#39;emp&#39;;</pre><p>To retrieve the variable from hiveconf, you have to explicitly specify the namespace. Not specifying the namespace returns an error.</p><pre>0: jdbc:hive2://127.0.0.1:10000&gt;SELECT * FROM ${hiveconf:table}<br># Below statement returns an error<br>0: jdbc:hive2://127.0.0.1:10000&gt;SELECT * FROM ${table}</pre><h4>Hive Custom Variables (hivevar)</h4><p>Hive version 0.8.0 introduced a new namespace hivevar to set the custom variables (JIRA <a href="https://issues.apache.org/jira/browse/HIVE-2020">HIVE-2020</a>), this separates custom variables from Hive default config variables. Hive recommends using hivevar explicitly for custom variables.</p><p>The below example sets the emp value to the table variable in hivevar namespace.</p><pre>0: jdbc:hive2://127.0.0.1:10000&gt;SET hivevar:table=&#39;emp&#39;;</pre><p>To retrieve values from hivevar namespace, you can either specify hivevar namespace or ignore it as hivevar is a default namespace for retrieval.</p><pre>0: jdbc:hive2://127.0.0.1:10000&gt;SELECT * FROM ${table}<br>(or)<br>0: jdbc:hive2://127.0.0.1:10000&gt;SELECT * FROM ${hivevar:table}</pre><h4>Hive System Variables (system)</h4><p>Hive default provides certain system variables and all system variables can be accessed in Hive using system namespace. For example.</p><pre>system:java.version<br>system:user.timezone<br>system:user.home</pre><h4>Hive Environment Variables (env)</h4><p>Hive also default provides certain environment variables and all environment variables can be accessed in Hive using env namespace. For example</p><pre>env:PWD<br>env:USER</pre><h3>3. Set Variables in Hive Scripts</h3><p>Hive scripts support using all variables explained above, you can use any of these along with their namespace.</p><p>let’s create test.hql file with the below content. If you notice, I am referring to the table name from hivevar namespace.</p><pre>0: jdbc:hive2://127.0.0.1:10000&gt;SELECT * FROM ${hivevar:table};</pre><p>Execute the test.hql script by running the below command.</p><pre>bin/beeline -u jdbc:hive2://127.0.0.1:10000 scott tiger -f test.hql<br>(or)<br>bin/hive -f test.hql</pre><p>Before executing the queries mentioned in the script, hive replaces the variables with the actual values from the namespace during the query construct stage.</p><h3>4. How to set and pass command line values to Hive scripts?</h3><p>Hive also supports setting a variable from the command line when <a href="https://sparkbyexamples.com/apache-hive/connect-to-hive-using-beeline/">starting a Hive CLI or beeline</a>.</p><pre>bin/beeline -u jdbc:hive2://127.0.0.1:10000 scott tiger --hivevar table=&#39;emp&#39;<br>bin/hive --hivevar table=&#39;emp&#39;</pre><p>You can refer to this variable in Hive CLI.</p><pre>0: jdbc:hive2://127.0.0.1:10000&gt;SELECT &amp; FROM ${hivevar:table};</pre><p>You can also call test.hql script by setting command line variables.</p><pre>bin/beeline -u jdbc:hive2://127.0.0.1:10000 scott tiger --hivevar table=&#39;dept&#39; -f test.hql<br>(or)<br>bin/hive --hivevar table=&#39;dept&#39; -f test.hql</pre><p>This should select data from the dept table.</p><h3>5. List all Hive variables</h3><p>From Hive scripts you can access the environment (env), system, hive configuration, and custom variables. since there are many variables it’s hard to remember all of them, use a set to list variables in the hive terminal</p><pre>0: jdbc:hive2://127.0.0.1:10000&gt;set;<br>(or)<br>0: jdbc:hive2://127.0.0.1:10000&gt;set -v; #This lists all variables with namespace</pre><p>This set command lists all available variables and configurations in Hive.</p><h3>6. How variable substitute works?</h3><p>Hive substitutes the value for a variable when a query is constructed with the variable.</p><ul><li>Values on Hive variables are visible only to active sessions where it’s been assigned and they cannot be accessible from another session.</li></ul><h3>7. Disabling Variable Substitution</h3><p>By default Hive substitutes all variables, you can disable these using (<a href="https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-hive.variable.substitute">hive.variable.substitute</a>=true) in case you want to run a script without substitution variables.</p><pre>0: jdbc:hive2://127.0.0.1:10000&gt;set hive.variable.substitute=false;</pre><p>I hope this helps!!</p><p>Happy Learning !!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=17a412146087" width="1" height="1" alt=""><hr><p><a href="https://medium.com/sparkbyexamples/do-you-know-you-can-use-variables-in-hive-17a412146087">Do you know you can use Variables in Hive?</a> was originally published in <a href="https://medium.com/sparkbyexamples">SparkByExamples</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Union Pandas DataFrames?]]></title>
            <link>https://medium.com/sparkbyexamples/how-to-union-pandas-dataframes-a9b0736495d9?source=rss----fe7db074b243---4</link>
            <guid isPermaLink="false">https://medium.com/p/a9b0736495d9</guid>
            <category><![CDATA[data-engineer]]></category>
            <category><![CDATA[pandas-dataframe]]></category>
            <category><![CDATA[big-data-analytics]]></category>
            <category><![CDATA[python-programming]]></category>
            <dc:creator><![CDATA[Naveen Nelamali]]></dc:creator>
            <pubDate>Fri, 12 Jan 2024 05:53:05 GMT</pubDate>
            <atom:updated>2024-01-12T05:54:34.793Z</atom:updated>
            <content:encoded><![CDATA[<h3>How to Union Pandas DataFrames by Vertical &amp; Horizontal</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ka8c1HXIMTMd1G6wsMWF4g.png" /></figure><p>In pandas, you can use the concat() function to concatenate or union the DataFrames along with a particular axis (either rows or columns). To union the Pandas DataFrames use the concat() function, by either vertical concatenation (concatenating along rows) or horizontal concatenation (concatenating along columns).</p><p>In this article, I will explain how to union two pandas DataFrames by rows and columns with examples.</p><h3>Create Two DataFrames</h3><p>Let’s create two DataFrames with a few rows and columns.</p><pre># Create DataFrame<br>import pandas as pd<br>technologies = [ [&quot;Spark&quot;,20000,&quot;30days&quot;], <br>                 [&quot;Pandas&quot;,25000,&quot;40days&quot;], <br>               ]<br>column_names=[&quot;Courses&quot;,&quot;Fee&quot;,&#39;Duration&#39;]<br>df=pd.DataFrame(technologies, columns=column_names)<br>print(&quot;First DataFrame:\n&quot;,df)<br><br># Create second DataFrame<br>technologies = [ [&quot;Hadoop&quot;,25000,&quot;50days&quot;], <br>                 [&quot;Java&quot;,30000,&quot;40days&quot;], <br>               ]<br>column_names=[&quot;Courses&quot;,&quot;Fee&quot;,&#39;Duration&#39;]<br>df1=pd.DataFrame(technologies, columns=column_names)<br>print(&quot;Second DataFrame:\n&quot;,df1)</pre><p>Yields below output</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/260/0*z9tLb95kQwEIBomq" /></figure><h3>Vertical Union of Pandas DataFrames (Concatenation along Rows)</h3><p>To concatenate (union) pandas DataFrames using the concat() method to concat two DataFrames by rows meaning appending two DataFrames. By default, it performs append operations similar to a union where it bright all rows from both DataFrames to a single DataFrame.</p><p>In the below example, concatenates the DataFrames df and df1 along the rows (axis=0), performing a union operation. The resulting DataFrame (df2) contains all the rows from both original DataFrames, and the index is not reset, meaning it retains the original indices.</p><pre># Union pandas dataframes using concat<br>df2 = pd.concat([df, df1])<br>print(&quot;Union pandas dataframes using concat:\n&quot;, df2)</pre><p>Yields below output.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/305/0*4poS66cmA3OFDvl_" /></figure><h3>Concatenating Along Rows (axis=0)</h3><p>If you want to perform a union operation on two Pandas DataFrames using concat(), you can concatenate them along the rows (axis=0). For instance, ignore_index=True is used to reset the index of the resulting DataFrame.</p><pre># Concatenate along rows (performing a union)<br>df2 = pd.concat([df, df1], ignore_index=True)<br>print(&quot;Union pandas DataFrames and reset index:\n&quot;, df2)<br><br># Output:<br>#Union pandas DataFrames and reset index:<br>#   Courses    Fee Duration<br>#0   Spark  20000   30days<br>#1  Pandas  25000   40days<br>#2  Hadoop  25000   50days<br>#3    Java  30000   40days</pre><p>As you can see, the resulting DataFrame includes all unique rows from both original DataFrames, performing a union operation. If there are duplicate rows, they will be retained in the result.</p><h3>5. Horizontal Union (Concatenation along Columns)</h3><p>Alternatively, if you want to concatenate DataFrames along columns, you can use axis=1 as an argument in the pd.concat() function.</p><pre># Concatenate pandas DataFrames along columns<br>df2 = pd.concat([df, df1], axis=1)<br>print(&quot;Concatenate pandas DataFrames along columns:\n&quot;, df2)<br><br># Output<br>#Concatenate pandas DataFrames along columns:<br>#   Courses    Fee Duration Courses    Fee Duration<br>#0   Spark  20000   30days  Hadoop  25000   50days<br>#1  Pandas  25000   40days    Java  30000   40days</pre><h3>6. Concatenate with keys</h3><p>When you use the keys parameter with pd.concat(), it allows you to create a hierarchical index based on the provided keys.</p><p>In the below example, the resulting DataFrame has a multi-level index where the first level corresponds to the keys (‘df1’ and ‘df2’), and the second level corresponds to the original index of the DataFrames.</p><pre># Concatenate with keys<br>result = pd.concat([df, df1], keys=[&#39;df&#39;, &#39;df1&#39;])<br>print(&quot;Concatenate with keys:\n&quot;,result)<br><br># Union with keys<br>result = pd.concat([df, df1], keys=[&#39;df&#39;, &#39;df1&#39;]).drop_duplicates()<br>print(&quot;Union with keys:\n&quot;,result)<br><br># Output:<br># Concatenate with keys:<br>#       Courses    Fee Duration<br># df  0   Spark  20000   30days<br>#     1  Pandas  25000   40days<br># df1 0  Hadoop  25000   50days<br>#     1    Java  30000   40days</pre><p>Hope you like it.</p><p>Happy Learning !!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a9b0736495d9" width="1" height="1" alt=""><hr><p><a href="https://medium.com/sparkbyexamples/how-to-union-pandas-dataframes-a9b0736495d9">How to Union Pandas DataFrames?</a> was originally published in <a href="https://medium.com/sparkbyexamples">SparkByExamples</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why Avoid UDFs in Spark & PySpark?]]></title>
            <link>https://medium.com/sparkbyexamples/why-avoid-udfs-in-spark-pyspark-e5903050e33c?source=rss----fe7db074b243---4</link>
            <guid isPermaLink="false">https://medium.com/p/e5903050e33c</guid>
            <category><![CDATA[apache-spark]]></category>
            <category><![CDATA[spark-udf]]></category>
            <category><![CDATA[pyspark-udf]]></category>
            <dc:creator><![CDATA[Naveen Nelamali]]></dc:creator>
            <pubDate>Fri, 12 Jan 2024 04:54:05 GMT</pubDate>
            <atom:updated>2024-01-12T05:06:17.595Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EmphAOLyQ0YWbt9Wnrx0IQ.jpeg" /></figure><p>User-Defined Functions (UDFs) are a powerful feature in Apache Spark and PySpark that allow users to define their own custom functions to perform complex data operations. However, while UDFs can be useful in certain situations, they can also be a source of performance issues and other problems in Spark applications. In this article, we will explore what makes UDFs problematic in Spark and PySpark, and what alternatives exist for performing data operations that can help overcome these issues.</p><h3>What is Spark UDF and Why do we need it?</h3><p>UDFs are custom functions written by users that can be applied to data in a Spark/PySpark DataFrame or RDD. They can be helpful when working with complex data types or performing non-standard operations that are not supported by Spark’s built-in functions.</p><p>Below is an example of UDF in Spark with Scala</p><pre>import org.apache.spark.sql.functions.udf<br>import org.apache.spark.sql.functions.col<br>import org.apache.spark.sql.{Row, SparkSession}<br><br>object SparkUDF extends App{<br><br>  //Create spark session object<br><br>  import spark.implicits._<br>  val columns = Seq(&quot;Seqno&quot;,&quot;Quote&quot;)<br>  val data = Seq((&quot;1&quot;, &quot;Be the change that you wish to see in the world&quot;))<br>  val df = data.toDF(columns:_*)<br><br>  // Create UDF<br>  val convertCase =  (str:String) =&gt; {<br>    val arr = str.split(&quot; &quot;)<br>    arr.map(f=&gt;  f.substring(0,1).toUpperCase + f.substring(1,f.length)).mkString(&quot; &quot;)<br>  }<br>  val convertUDF = udf(convertCase)<br><br>  // Use UDF on DataFrame<br>  df.select(col(&quot;Seqno&quot;),<br>    convertUDF(col(&quot;Quote&quot;)).as(&quot;Quote&quot;) ).show(false)<br>}</pre><h3>Why using UDFs are Bottle-neck to Performance</h3><p>User-Defined Functions (UDFs) in Spark can incur performance issues due to serialization overhead, necessitating the conversion of data between internal and external representations. Additionally, UDFs may trigger data movement across nodes, introducing network overhead.</p><p>Spark struggles to optimize complex UDF logic, leading to suboptimal execution plans and reduced performance. Manual resource management for UDFs can be challenging, impacting overall job efficiency.</p><h3>Why to Avoid UDFs in PySpark</h3><p>One of the main issues with UDFs is that they can be slow to execute. Since UDFs are written in Python, they are not compiled like PySpark’s built-in functions, which are implemented in Java. This means that UDFs can be significantly slower when processing large datasets, especially when compared to native PySpark functions. Additionally, UDFs can be memory-intensive, which can lead to out-of-memory errors when working with large datasets.</p><h3>Disadvantages of UDF</h3><p>While User-Defined Functions (UDFs) in Spark offer flexibility and extensibility, there are some disadvantages associated with their usage. Here are some drawbacks of using UDFs in Spark:</p><p><strong>Performance Overhead:</strong></p><ul><li>UDFs introduce some performance overhead compared to built-in Spark functions. The execution of UDFs involves the cost of serialization, data transfer, and deserialization.</li></ul><p><strong>Limited Optimization Opportunities:</strong></p><ul><li>Spark might not be able to optimize UDFs as effectively as built-in functions. This could lead to suboptimal execution plans and, consequently, slower performance.</li></ul><p><strong>Type Safety and Error Handling:</strong></p><ul><li>UDFs might not provide the same level of type safety and error handling as built-in functions. Incorrect data types or errors within UDFs can lead to runtime issues that are harder to debug.</li></ul><p><strong>Data Movement Overhead:</strong></p><ul><li>When using UDFs, there could be additional data movement between nodes in a distributed environment, leading to increased network overhead.</li></ul><p><strong>Resource Management:</strong></p><ul><li>Managing resources (memory, CPU) for UDFs might require manual tuning, and improper resource allocation could impact overall job performance.</li></ul><p><strong>Limited Language Support:</strong></p><ul><li>UDFs are typically written in languages like Scala, Java, or Python. While this covers a broad range of use cases, it might not be as versatile as using native Spark functions for certain operations.</li></ul><p><strong>Dependency Management:</strong></p><ul><li>UDFs might have external dependencies that need to be managed separately. Ensuring that these dependencies are available across all worker nodes can be challenging.</li></ul><p><strong>Debugging Complexity:</strong></p><ul><li>Debugging UDFs can be more challenging than debugging Spark SQL or DataFrame operations. The development and debugging process might be less user-friendly.</li></ul><p><strong>Compatibility with Spark Ecosystem:</strong></p><ul><li>Some components of the Spark ecosystem, such as Spark Streaming or Spark MLlib, might not seamlessly support UDFs, limiting their use in certain contexts.</li></ul><p><strong>Potential for Non-Optimized Execution Plans:</strong></p><ul><li>Depending on the complexity of the UDF logic, Spark might struggle to generate optimized execution plans, leading to inefficient query execution.</li></ul><p>Despite these disadvantages, UDFs remain a valuable tool for extending Spark’s functionality when necessary. Careful consideration of when and how to use UDFs is essential to mitigate these drawbacks and ensure optimal performance.</p><h3>Conclusion</h3><p>In conclusion, UDFs can be helpful when working with complex data types or performing non-standard operations in PySpark. However, they come with a number of downsides that can make them problematic in larger-scale applications.</p><p>To overcome these issues, it is important to consider using PySpark’s built-in functions, SQL functions, and libraries whenever possible. These alternatives are optimized for performance and are easier to distribute and debug than UDFs, making them a more practical choice for large-scale data processing applications.</p><p>Happy Learning !!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e5903050e33c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/sparkbyexamples/why-avoid-udfs-in-spark-pyspark-e5903050e33c">Why Avoid UDFs in Spark &amp; PySpark?</a> was originally published in <a href="https://medium.com/sparkbyexamples">SparkByExamples</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What are the Specific Roles of Spark Driver and Executor]]></title>
            <link>https://medium.com/sparkbyexamples/what-are-the-specific-roles-of-spark-driver-and-executor-a17f90c7e7fd?source=rss----fe7db074b243---4</link>
            <guid isPermaLink="false">https://medium.com/p/a17f90c7e7fd</guid>
            <category><![CDATA[pyspark]]></category>
            <category><![CDATA[apache-spark-executor]]></category>
            <category><![CDATA[apache-spark]]></category>
            <category><![CDATA[apache-spark-driver]]></category>
            <dc:creator><![CDATA[Naveen Nelamali]]></dc:creator>
            <pubDate>Fri, 12 Jan 2024 04:26:43 GMT</pubDate>
            <atom:updated>2024-01-12T19:24:05.216Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nJum-35aJ-9jEVR8bWPKMA.jpeg" /></figure><p>Have you ever wondered what the different roles of Apache Spark Driver and Executor play when running your application in a distributed cluster?</p><p>Apache Spark is a popular open-source distributed computing system that is widely used for big data processing and analytics. When running Spark applications, two crucial components of the Spark architecture are the Spark Driver and Executor.</p><p>While both are responsible for managing tasks and resources, they have distinct roles and responsibilities in the overall execution of a Spark job. In this article, we will explore the specific roles of the Spark Driver and Executor, and how they work together to execute Spark applications efficiently and effectively.</p><h3>What is the role of Spark Driver?</h3><p>The Apache Spark Driver is the program that declares the SparkContext, which is responsible for converting the user program into a series of tasks that can be distributed across the cluster. It also coordinates the execution of tasks and communicates with the Cluster Manager to allocate resources for the application. In short, the Spark Driver plays a crucial role in managing the overall execution of a Spark application.</p><p><strong>Spark Driver Configurations</strong></p><p>You need to specify the driver configurations when submitting the spark application using spark-submit command. Following are some of them you can use based on your needs. For a complete list, refer to the Spark official documentation.</p><ul><li>--driver-memory: Specifies the amount of memory to allocate for the driver (e.g., &quot;1g&quot; for 1 gigabyte).</li><li>--driver-cores: Specifies the number of cores to allocate for the driver.</li><li>--driver-class-path: Specifies the classpath for the driver.</li><li>--driver-java-options: Specifies extra Java options for the driver.</li><li>--driver-library-path: Specifies the library path for the driver.</li><li>--conf spark.driver.maxResultSize: Limits the maximum size of results that Spark will return to the driver.</li><li>--conf spark.driver.host: Specifies the hostname or IP address the driver runs on.</li></ul><h3>What is the role of Spark Executor?</h3><p>In Apache Spark, an Executor is a process that is responsible for running the tasks in parallel on worker nodes in the cluster. The Spark driver program launches the Executor process, and it runs on the worker nodes to execute the tasks the driver assigns. Hence, it is critical to understand the difference between Spark Driver and Executor.</p><p>The Executor is a key component of the Spark runtime architecture because it is responsible for processing the data and executing the code in parallel on the worker nodes. The Executor runs the user-defined Spark code, which can be written in various programming languages such as Scala, Java, Python or R, and it performs the necessary calculations and transformations on the data using the RDD (Resilient Distributed Dataset) API or the DataFrame API.</p><p><strong>Spark Executor Configurations</strong></p><ul><li>--executor-memory: Specifies the amount of memory to allocate per executor (e.g., &quot;1g&quot; for 1 gigabyte).</li><li>--num-executors: Specifies the number of executors to launch.</li><li>--executor-cores: Specifies the number of cores to allocate for each executor.</li><li>--conf spark.executor.extraClassPath: Specifies extra classpath entries for executors.</li><li>--conf spark.executor.extraJavaOptions: Specifies extra Java options for executors.</li><li>--conf spark.executor.extraLibraryPath: Specifies extra library path entries for executors.</li><li>--conf spark.yarn.executor.memoryOverhead: Specifies the amount of non-heap memory to be allocated per executor.</li><li>--conf spark.executor.memoryFraction: Specifies the fraction of the heap space that is allocated for Spark&#39;s memory management.</li><li>--conf spark.dynamicAllocation.enabled: Enables or disables dynamic allocation of executors.</li></ul><p><strong>Spark Executor is a fault-tolerant</strong></p><p>The Executor is designed to be fault-tolerant and resilient to failures, which is essential for handling large-scale data processing workloads. If an executor fails due to hardware or software issues, the Spark framework automatically re-launches the Executor on a different worker node and re-runs the failed tasks to ensure that the processing is not interrupted.</p><p>The number of Executors and their configuration parameters such as memory, cores, and parallelism can be adjusted based on the specific requirements of the Spark application and the resources available in the cluster. The optimal configuration can help to improve the performance and scalability of the Spark application and allow it to process large amounts of data efficiently.</p><h3>Conclusion</h3><p>In summary, the Spark Driver is the central control and coordination point for a Spark application, managing the overall execution, while Spark Executors are responsible for executing specific tasks on worker nodes, each managing its own memory space.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a17f90c7e7fd" width="1" height="1" alt=""><hr><p><a href="https://medium.com/sparkbyexamples/what-are-the-specific-roles-of-spark-driver-and-executor-a17f90c7e7fd">What are the Specific Roles of Spark Driver and Executor</a> was originally published in <a href="https://medium.com/sparkbyexamples">SparkByExamples</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Remember These when Accessing Matrix Elements in R]]></title>
            <link>https://medium.com/sparkbyexamples/accessing-matrix-elements-in-r-e278fa225cfd?source=rss----fe7db074b243---4</link>
            <guid isPermaLink="false">https://medium.com/p/e278fa225cfd</guid>
            <dc:creator><![CDATA[Naveen Nelamali]]></dc:creator>
            <pubDate>Thu, 11 Jan 2024 22:32:36 GMT</pubDate>
            <atom:updated>2023-11-11T15:57:09.036Z</atom:updated>
            <content:encoded><![CDATA[<p>When accessing elements in an R matrix, there are several important things to remember to ensure accurate and effective data manipulation:</p><p><strong>Indexing:</strong></p><p>R uses 1-based indexing, meaning the first element of a matrix is accessed with the index 1, not 0. Keep this in mind to avoid off-by-one errors.</p><pre># Correct indexing<br>element &lt;- my_matrix[1, 2]  # Accesses the element in the first row, second column<br><br># Incorrect indexing<br>element &lt;- my_matrix[0, 1]  # This will result in an error</pre><p><strong>Comma Separation for Rows and Columns:</strong></p><p>When accessing elements, use a comma to separate row and column indices. For example, my_matrix[1, 2] refers to the element in the first row and second column.</p><p><strong>Single Square Brackets for Single Elements:</strong></p><p>Use single square brackets ([]) when accessing individual elements. The result is a single value.</p><pre># Accesses a single element<br>element &lt;- my_matrix[1, 2]  </pre><p><strong>Double Square Brackets for Entire Rows or Columns:</strong> If you want to extract entire rows or columns, use double square brackets ([[]]). This returns a vector.</p><pre># Extracts the entire first row<br>row_vector &lt;- my_matrix[1, ]<br><br># Extracts the entire second column  <br>col_vector &lt;- my_matrix[, 2]  </pre><p><strong>Using </strong><strong>: for Sequences:</strong> You can use the colon (:) operator to create sequences of indices when accessing multiple elements.</p><pre># Accesses elements in rows 1 through 3 of the second column<br>elements &lt;- my_matrix[1:3, 2]  </pre><p><strong>Logical Indexing:</strong> You can use logical vectors to subset a matrix based on conditions.</p><pre># Selects rows where the first column value is greater than 3<br>subset_matrix &lt;- my_matrix[my_matrix[, 1] &gt; 3, ] </pre><p><strong>Matrix Attributes:</strong> Be aware of any row or column names assigned to the matrix. You can use these names for indexing in addition to numeric indices.</p><pre># Accesses the element in the row named &quot;Row1&quot; and column named &quot;Column2&quot;<br>element &lt;- my_matrix[&quot;Row1&quot;, &quot;Column2&quot;] </pre><p>By keeping these points in mind, you can avoid common indexing errors and efficiently access and manipulate elements within R matrices.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e278fa225cfd" width="1" height="1" alt=""><hr><p><a href="https://medium.com/sparkbyexamples/accessing-matrix-elements-in-r-e278fa225cfd">Remember These when Accessing Matrix Elements in R</a> was originally published in <a href="https://medium.com/sparkbyexamples">SparkByExamples</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Which Spark HBase Connector to use in 2019? — Spark by {Examples}]]></title>
            <link>https://medium.com/sparkbyexamples/which-spark-hbase-connector-to-use-in-2019-spark-by-examples-792f749a7e8c?source=rss----fe7db074b243---4</link>
            <guid isPermaLink="false">https://medium.com/p/792f749a7e8c</guid>
            <category><![CDATA[apache-spark]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[hadoop]]></category>
            <category><![CDATA[apache-hbase]]></category>
            <dc:creator><![CDATA[Naveen Nelamali]]></dc:creator>
            <pubDate>Thu, 11 Jan 2024 22:32:14 GMT</pubDate>
            <atom:updated>2022-12-20T19:56:10.444Z</atom:updated>
            <content:encoded><![CDATA[<h3>Which Spark HBase Connector to use in 2019?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XHQcyexSIHQChzTnu63pTQ.png" /></figure><p>This tutorial explains different Spark connectors and libraries to interact with HBase Database and provides a Hortonworks connector example of how to create DataFrame from and Insert DataFrame to the table.</p><p>On the internet, you would find several ways and API’s to connect Spark to HBase and some of these are outdated or not maintained properly. Here, I will explain some libraries and what they are used for and later will see some spark SQL examples.</p><p>Let’s see these in detail.</p><h3>Apache HBase Client</h3><p>Apache hbase-client API comes with HBase distribution and you can find this jar in /lib at your installation directory. All Spark connectors use this library to interact with database natively. Below is maven dependency to use.</p><pre>&lt;dependency&gt; &lt;groupId&gt;org.apache.hbase&lt;/groupId&gt; &lt;artifactId&gt;hbase-client&lt;/artifactId&gt; &lt;version&gt; replace hbase version &lt;/version&gt; &lt;/dependency&gt;</pre><p>If you want to connect to HBase from Java or Scala to connect to HBase, you can use this client API directly without any third-party libraries but with Spark you need connectors. You can find the recent releases and it’s compatible dependencies at Maven repository.</p><p>When you need to connect to HBase, you don’t have to add this dependency as connectors provide this library as a transitive dependency. and providing this additionally would result in incompatibility issues.</p><h3>Spark HBase Connector ( hbase-spark )</h3><p>hbase-spark API enables us to integrate Spark and fulfill the gap between Key-Value structure and Spark SQL table structure, and enables users to perform complex data analytical work on top of HBase.</p><p>It also helps us to leverage the benefits of RDD and DataFrame to use. HBaseContext is the root of all Spark integration, the HBaseContext reads HBase configurations and pushes them to the executors. This allows us to have an HBase Connection per Spark Executor.</p><p>Using this we should able to do parallel processing like bulk inserts through bulkPut, bulk deletes through buldDelete, build retrieval using bulkGet functions and it also has more functions like mapPartition e.t.c, All these functions return RDD and to use these we should import &quot;org.apache.hadoop.hbase.spark.HBaseRDDFunctions._&quot;</p><p>When using this with DataFrame, use “ org.apache.hadoop.hbase.spark &quot; as a DataSource</p><pre>&lt;dependency&gt; &lt;groupId&gt;org.apache.hbase.connectors.spark&lt;/groupId&gt; &lt;artifactId&gt;hbase-spark&lt;/artifactId&gt; &lt;version&gt;replace version here&lt;/version&gt; &lt;/dependency&gt;</pre><p>You can find the recent releases and it’s compatible dependencies at Maven repository.</p><p>Note that this has been published to two maven artifacts, first “ org.apache.hbase&quot; doesn&#39;t have new changes on &quot;central&quot; table as of writing this post and all versions released were beta and it has released from Cloudera and Hortonworks with latest releases. In 2019 second artifact &quot; org.apache.hbase.connectors.spark &quot; created and released the first version with the latest version supported.</p><h3>Spark Hortonworks Connector ( shc-core )</h3><p>shc-core is from Hortonworks which provides DataSource &quot; org.apache.spark.sql.execution.datasources.hbase &quot; to integrate DataFrame with HBase, and it uses &quot;Spark HBase connector&quot; as dependency hence, we can use all its operations we discussed in the previous section.</p><pre>&lt;dependency&gt; &lt;groupId&gt;com.hortonworks&lt;/groupId&gt; &lt;artifactId&gt;shc-core&lt;/artifactId&gt; &lt;version&gt;replace version here&lt;/version&gt; &lt;/dependency&gt;</pre><p>You can find the recent releases and it’s compatible dependencies at Maven repository.</p><h3>DataFrame operations on HBase Table</h3><p>Regardless of which Connector implementation you use, the underlying usage would not change much hence, In this remaining tutorial, we will learn with examples on how to use the HBase from Hortonworks.</p><p><strong>Prerequisites</strong>:</p><p><a href="https://sparkbyexamples.com/hbase/setup-run-hbase-standalone-mode/">HBase installation and setup at least Standalone Mode</a></p><p>We should define Catalog for the schema mapping between HBase table and Spark DataFrame, prepare the data and populate the table, then load it to DataFrame. After that, users can do integrated query and access records in HBase tables with SQL query. The following illustrates the basic procedure.</p><h3>Create an HBase table</h3><p>Use create command to create the table with at least one column family. Below statement creates &#39;employee&#39; table with &#39;person&#39; and &#39;address&#39; column family.</p><p>Related: <a href="https://sparkbyexamples.com/hbase/hbase-create-table/">How to create an HBase table using shell</a></p><pre>Hbase&gt; create &#39;employee&#39;, &#39;person&#39;, &#39;address&#39;</pre><h3>Define Catalog Schema</h3><p>Define catalog schema to map HBase table to Spark DataFrame. Below schema specifies to create a table ‘employee’ in ‘default’ name schema with columns ‘key’, ‘firstName’, ‘lastName’, ‘middleName’ in ‘person’ column family and ‘addressLine1’, ‘city’, ‘state’ and ‘zipCode’ in ‘address’ column family. It’s not necessary to have a column names same between DataFrame and table hence, here we have used different column names for better understanding. For example, fName is DataFrame name for firstName in the table.</p><p>Note that the rowkey also has to be defined in details as a column (key), which has a specific column family rowkey</p><pre>defcatalog= s&quot;&quot;&quot;{ |&quot;table&quot;:{&quot;namespace&quot;:&quot;default&quot;,&quot;name&quot;:&quot;employee&quot;}, |&quot;rowkey&quot;:&quot;key&quot;, |&quot;columns&quot;:{ |&quot;key&quot;:{&quot;cf&quot;:&quot;rowkey&quot;,&quot;col&quot;:&quot;key&quot;,&quot;type&quot;:&quot;string&quot;}, |&quot;fName&quot;:{&quot;cf&quot;:&quot;person&quot;,&quot;col&quot;:&quot;firstName&quot;,&quot;type&quot;:&quot;string&quot;}, |&quot;lName&quot;:{&quot;cf&quot;:&quot;person&quot;,&quot;col&quot;:&quot;lastName&quot;,&quot;type&quot;:&quot;string&quot;}, |&quot;mName&quot;:{&quot;cf&quot;:&quot;person&quot;,&quot;col&quot;:&quot;middleName&quot;,&quot;type&quot;:&quot;string&quot;}, |&quot;addressLine&quot;:{&quot;cf&quot;:&quot;address&quot;,&quot;col&quot;:&quot;addressLine&quot;,&quot;type&quot;:&quot;string&quot;}, |&quot;city&quot;:{&quot;cf&quot;:&quot;address&quot;,&quot;col&quot;:&quot;city&quot;,&quot;type&quot;:&quot;string&quot;}, |&quot;state&quot;:{&quot;cf&quot;:&quot;address&quot;,&quot;col&quot;:&quot;state&quot;,&quot;type&quot;:&quot;string&quot;}, |&quot;zipCode&quot;:{&quot;cf&quot;:&quot;address&quot;,&quot;col&quot;:&quot;zipCode&quot;,&quot;type&quot;:&quot;string&quot;} |} |}&quot;&quot;&quot;.stripMargin</pre><p>Create DataFrame</p><p>Now, Let’s create data and DataFrame from it to work with.</p><pre>Val data=Seq(Employee(&quot;1&quot;,&quot;Abby&quot;,&quot;Smith&quot;,&quot;K&quot;,&quot;3456main&quot;,&quot;Orlando&quot;,&quot;FL&quot;,&quot;45235&quot;), Employee(&quot;2&quot;,&quot;Amaya&quot;,&quot;Williams&quot;,&quot;L&quot;,&quot;123Orange&quot;,&quot;Newark&quot;,&quot;NJ&quot;,&quot;27656&quot;), Employee(&quot;3&quot;,&quot;Alchemy&quot;,&quot;Davis&quot;,&quot;P&quot;,&quot;Warners&quot;,&quot;Sanjose&quot;,&quot;CA&quot;,&quot;34789&quot;)) Val spark:SparkSession=SparkSession.builder() .master(&quot;local[1]&quot;) .appName(&quot;SparkByExamples.com&quot;) .getOrCreate() Import spark.implicits._ Val df=spark.sparkContext.parallelize(data).toDF</pre><h3>Insert DataFrame Rows to table</h3><p>As described earlier, Hortonworks connecters define DataSource “org.apache.spark.sql.execution.datasources.hbase” to read from and write to the table. One, you have a DataFrame, let’s inserts it’s rows to the table using this DataSource in format function. write is an object of DataFrameWriter to insert the DataFrame rows to HBase storage systems. Given a DataFrame with specified schema catalog, save function will create an table with 5 regions and save the DataFrame inside.</p><pre>df.write.options( Map(HBaseTableCatalog.tableCatalog-&gt;catalog,HBaseTableCatalog.newTable-&gt;&quot;4&quot;)) .format(&quot;org.apache.spark.sql.execution.datasources.hbase&quot;) .save()</pre><p>scan ‘employee’ shell command returns the following with 3 row keys 1,2,3</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*RM9ySVnafrE0T0CH" /></figure><h3>Reading and applying filters</h3><p>Along with this, we should pass the table name to scan and catalog for DataFrame structure as options.</p><pre>Val hbaseDF=spark.read .options(Map(HBaseTableCatalog.tableCatalog-&gt;catalog)) .format(&quot;org.apache.spark.sql.execution.datasources.hbase&quot;) .load() hbaseDF.printSchema() hbaseDF.show(false)</pre><p>printSchema() prints the following.</p><pre>root |-- key: string (nullable = true) |-- fName: string (nullable = true) |-- lName: string (nullable = true) |-- mName: string (nullable = true) |-- addressLine: string (nullable = true) |-- city: string (nullable = true) |-- state: string (nullable = true) |-- zipCode: string (nullable = true)</pre><p>show(false) returns the following. false arguments say not to truncate the data on results.</p><pre>+---+-------+--------+-----+-----------+-------+-----+-------+ |key|fName |lName |mName|addressLine|city |state|zipCode| +---+-------+--------+-----+-----------+-------+-----+-------+ |1 |Abby |Smith |K |3456 main |Orlando|FL |45235 | |2 |Amaya |Williams|L |123 Orange |Newark |NJ |27656 | |3 |Alchemy|Davis |P |Warners |Sanjose|CA |34789 | +---+-------+--------+-----+-----------+-------+-----+-------+</pre><p>Now, we should able to apply DataFrame operations on this.</p><pre>hbaseDF.filter($&quot;key&quot;===&quot;1&quot;&amp;&amp;$&quot;state&quot;===&quot;FL&quot;) .select(&quot;key&quot;,&quot;fName&quot;,&quot;lName&quot;) .show()</pre><h3>Running SQL Query on table</h3><p>Finally, Let’s create a temporary table on DataFrame which enables us to run SQL queries on the HBase table. The lifetime of this temporary table is tied to the SQLContext that was used to create.</p><pre>hbaseDF.createOrReplaceTempView(&quot;employeeTable&quot;) spark.sql(&quot;select*fromemployeeTablewherefName=&#39;Amaya&#39;&quot;).show</pre><p>Conclusion:</p><p>In this tutorial, you have learned different Spark HBase libraries to work with and the usage of the Hortonworks connector DataSource API with example. hope you like it.</p><p>Happy Learning 🙂</p><p><em>Originally published at </em><a href="https://sparkbyexamples.com/hbase/spark-hbase-connectors-which-one-to-use/"><em>https://sparkbyexamples.com</em></a><em> on August 30, 2019.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=792f749a7e8c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/sparkbyexamples/which-spark-hbase-connector-to-use-in-2019-spark-by-examples-792f749a7e8c">Which Spark HBase Connector to use in 2019? — Spark by {Examples}</a> was originally published in <a href="https://medium.com/sparkbyexamples">SparkByExamples</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Apache Spark and Parquet example — Spark by {Examples}]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/sparkbyexamples/apache-spark-and-parquet-example-spark-by-examples-5b701a9e606d?source=rss----fe7db074b243---4"><img src="https://cdn-images-1.medium.com/max/665/0*fUR5Kz6fS6DAyu1P" width="665"></a></p><p class="medium-feed-snippet">Parquet is a columnar file format that provides optimizations to speed up queries and is a far more efficient file format than CSV or JSON&#x2026;</p><p class="medium-feed-link"><a href="https://medium.com/sparkbyexamples/apache-spark-and-parquet-example-spark-by-examples-5b701a9e606d?source=rss----fe7db074b243---4">Continue reading on SparkByExamples »</a></p></div>]]></description>
            <link>https://medium.com/sparkbyexamples/apache-spark-and-parquet-example-spark-by-examples-5b701a9e606d?source=rss----fe7db074b243---4</link>
            <guid isPermaLink="false">https://medium.com/p/5b701a9e606d</guid>
            <category><![CDATA[spark-parquet-example]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[apache-spark]]></category>
            <category><![CDATA[spark-sql-on-parquet]]></category>
            <dc:creator><![CDATA[Naveen Nelamali]]></dc:creator>
            <pubDate>Thu, 11 Jan 2024 22:31:50 GMT</pubDate>
            <atom:updated>2022-12-20T19:57:45.071Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Spark “case when” and “when otherwise” usage — Spark by {Examples}]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-snippet">Like SQL &#x201C;case when&#x201D; statement, Spark also supports similar syntax using when otherwise or we can also use case when statement. So let&#x2019;s&#x2026;</p><p class="medium-feed-link"><a href="https://medium.com/sparkbyexamples/spark-case-when-and-when-otherwise-usage-spark-by-examples-e82259d62f91?source=rss----fe7db074b243---4">Continue reading on SparkByExamples »</a></p></div>]]></description>
            <link>https://medium.com/sparkbyexamples/spark-case-when-and-when-otherwise-usage-spark-by-examples-e82259d62f91?source=rss----fe7db074b243---4</link>
            <guid isPermaLink="false">https://medium.com/p/e82259d62f91</guid>
            <category><![CDATA[case-when]]></category>
            <category><![CDATA[dataframes]]></category>
            <category><![CDATA[spark-dataframe]]></category>
            <category><![CDATA[when-otherwise]]></category>
            <category><![CDATA[spark]]></category>
            <dc:creator><![CDATA[Naveen Nelamali]]></dc:creator>
            <pubDate>Thu, 11 Jan 2024 22:31:34 GMT</pubDate>
            <atom:updated>2019-08-08T06:38:17.003Z</atom:updated>
        </item>
    </channel>
</rss>