<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Janardhan Jai on Medium]]></title>
        <description><![CDATA[Stories by Janardhan Jai on Medium]]></description>
        <link>https://medium.com/@janardhanjai?source=rss-9f3df5a5f36b------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>Stories by Janardhan Jai on Medium</title>
            <link>https://medium.com/@janardhanjai?source=rss-9f3df5a5f36b------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 01:23:10 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@janardhanjai/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[PySpark Interview based Questions and Answers- Part -01]]></title>
            <link>https://medium.com/@janardhanjai/pyspark-interview-based-questions-and-answers-part-01-9b7ee05d6b51?source=rss-9f3df5a5f36b------2</link>
            <guid isPermaLink="false">https://medium.com/p/9b7ee05d6b51</guid>
            <dc:creator><![CDATA[Janardhan Jai]]></dc:creator>
            <pubDate>Mon, 13 Jan 2025 11:33:05 GMT</pubDate>
            <atom:updated>2025-01-13T11:37:50.479Z</atom:updated>
            <content:encoded><![CDATA[<ol><li><strong>How do you create a SparkSession in PySpark ?</strong></li></ol><p>Answer : To create SparkSession in PySpark we use the SparkSession.builder method.</p><p>Example :</p><blockquote><em>from pyspark.sql import SparkSession</em></blockquote><blockquote><em>spark = SparkSession.builder.appName(“appname”).getOrCreate()</em></blockquote><p>here ,</p><ul><li><strong>SparkSession.builder</strong> is used to configure the session.</li><li><strong>.appName(&quot;appname&quot;)</strong> sets the application name.</li><li><strong>.getOrCreate()</strong> either creates a new session or retrieves the existing one.</li></ul><p><strong>2) How to you read a CSV file into a Dataframe using Pyspark ?</strong></p><p>Answer: To read csv file into a Dataframe using Pyspark , we use the sparksession.read.csv method. we can also specify the options like header=True to treat the first row as header and inferschema=True to detectes the datatypes of the columns.</p><blockquote>from pyspark.sql from SparkSession</blockquote><blockquote>spark = SparkSession.builder.appName(“readcsvfile”).getOrCreate()</blockquote><blockquote>df = spark.read.csv(“<strong>path_to_your_file.csv</strong>”,header=True, inferschema=True)</blockquote><blockquote>df.show()</blockquote><ul><li><strong>SparkSession.builder</strong>: Initializes the SparkSession.</li><li><strong>spark.read.csv()</strong>: Reads the CSV file into a DataFrame.</li><li><strong>&quot;path_to_your_file.csv&quot;</strong>: Replace this with the path to your CSV file.</li><li><strong>header=True</strong>: Indicates that the first row of the CSV contains the header.</li><li><strong>inferSchema=True</strong>: Automatically infers the data types of the columns.</li><li><strong>df.show()</strong>: Displays the content of the DataFrame.</li></ul><p><strong>3)How do you show the schema of a DataFrame in PySpark?</strong></p><p>Answer: To display the schema of the dataframe in pyspark, we use the printschema() method.It shows the column names along with datatypes,which helps us to understand the structure of the dataframe.</p><blockquote>example:</blockquote><blockquote>from pyspark.sql import SparkSession</blockquote><blockquote># Initialize SparkSession<br>spark = SparkSession.builder \<br> .appName(“ShowSchemaExample”) \<br> .getOrCreate()</blockquote><blockquote># Example DataFrame<br>data = [(“Alice”, 30), (“Bob”, 25)]<br>columns = [“name”, “age”]</blockquote><blockquote># Create a DataFrame<br>df = spark.createDataFrame(data, columns)</blockquote><blockquote># Show the schema of the DataFrame<br>df.printSchema()</blockquote><p>Output:<br> |name: string <br> |age: integer</p><p><strong>4)How do we select Specific columns in Dataframe using PySpark ?</strong></p><p>Answer:To select specific columns in PySpark DataFrame, we use the method called select(). For example we select name and age columns. we call df.select(“name” , “age”).</p><p>Example:</p><blockquote>from pyspark.sql import SparkSession</blockquote><blockquote># Initialize SparkSession<br>spark = SparkSession.builder \<br> .appName(“SelectColumnsExample”) \<br> .getOrCreate()</blockquote><blockquote># Example DataFrame<br>data = [(“Alice”, 30, “Female”), (“Bob”, 25, “Male”)]<br>columns = [“name”, “age”, “gender”]</blockquote><blockquote># Create a DataFrame<br>df = spark.createDataFrame(data, columns)</blockquote><blockquote># Select specific columns (name and age)<br>selected_df = df.select(“name”, “age”)</blockquote><blockquote># Show the selected columns<br>selected_df.show()</blockquote><p>Output:<br>name age<br>Alice 30<br>Bob 25</p><p><strong>5) How do we Filter row based on a condition in pyspark dataframe ?</strong></p><p>Answer: To filter rows in a Pyspark Dataframe, we use the filter() method or where() method. For example, to filter rows where the age is greater than 30, we can use df.filter(df.age&gt;30).</p><p>from pyspark.sql import SparkSession</p><blockquote># Initialize SparkSession<br>spark = SparkSession.builder \<br> .appName(“FilterRowsExample”) \<br> .getOrCreate()</blockquote><blockquote># Example DataFrame<br>data = [(“Alice”, 30), (“Bob”, 25), (“Charlie”, 35)]<br>columns = [“name”, “age”]</blockquote><blockquote># Create a DataFrame<br>df = spark.createDataFrame(data, columns)</blockquote><blockquote># Filter rows where age is greater than 30<br>filtered_df = df.filter(df.age &gt; 35)</blockquote><blockquote># Show the filtered DataFrame<br>filtered_df.show()</blockquote><p>Output:<br> name age<br>Charlie 35</p><p><strong>6)How can we group by a column and perform an aggregation in PySpark?</strong></p><p>To group by a column and perform an aggregation in PySpark, we use the groupBy() method and then apply aggregation functions such as avg(), sum(), or count(). For example, to group by department and calculate the average and total salary, we use</p><p>example :</p><blockquote>from pyspark.sql import SparkSession<br>from pyspark.sql.functions import avg, sum</blockquote><blockquote># Initialize SparkSession<br>spark = SparkSession.builder \<br> .appName(“GroupByAggregationExample”) \<br> .getOrCreate()</blockquote><blockquote># Example DataFrame<br>data = [(“Alice”, “HR”, 5000), (“Bob”, “HR”, 6000), (“Charlie”, “IT”, 7000), (“David”, “IT”, 8000)]<br>columns = [“name”, “department”, “salary”]</blockquote><blockquote># Create a DataFrame<br>df = spark.createDataFrame(data, columns)</blockquote><blockquote># Group by department and calculate the average salary<br>grouped_df = df.groupBy(“department”).agg(<br> avg(“salary”).alias(“avg_salary”),<br> sum(“salary”).alias(“total_salary”)<br>)</blockquote><blockquote># Show the result<br>grouped_df.show()</blockquote><h3>Common Aggregations:</h3><ul><li><strong>avg(&quot;column&quot;)</strong>: Average of the specified column.</li><li><strong>sum(&quot;column&quot;)</strong>: Sum of the specified column.</li><li><strong>count(&quot;column&quot;)</strong>: Count of rows in each group.</li><li><strong>max(&quot;column&quot;)</strong>: Maximum value of the specified column.</li><li><strong>min(&quot;column&quot;)</strong>: Minimum value of the specified column.</li></ul><p><strong>7) what type of joins in PySpark ?</strong></p><h3>1. Inner Join (inner)</h3><p>An <strong>inner join</strong> returns only the rows where there is a match in both DataFrames. If there is no match, the row will not appear in the result.</p><p><strong>Syntax:</strong></p><pre>df1.join(df2, on=&quot;column_name&quot;, how=&quot;inner&quot;)</pre><p><strong>Example:</strong></p><pre># Only matching rows based on &#39;id&#39;<br>joined_df = df1.join(df2, on=&quot;id&quot;, how=&quot;inner&quot;)</pre><p><strong>Output:</strong></p><ul><li>Rows where the id exists in both df1 and df2 will be included in the result.</li></ul><h3>2. Left Join (left or left_outer)</h3><p>A <strong>left join</strong> returns all the rows from the left DataFrame (df1) and the matching rows from the right DataFrame (df2). If there is no match, the result will contain null for the columns from the right DataFrame.</p><p><strong>Syntax:</strong></p><pre>df1.join(df2, on=&quot;column_name&quot;, how=&quot;left&quot;)</pre><p><strong>Example:</strong></p><pre># All rows from df1, and matching rows from df2<br>left_joined_df = df1.join(df2, on=&quot;id&quot;, how=&quot;left&quot;)</pre><p><strong>Output:</strong></p><ul><li>All rows from df1 will be in the result. If there’s no match in df2, the corresponding columns will be null.</li></ul><h3>3. Right Join (right or right_outer)</h3><p>A <strong>right join</strong> returns all the rows from the right DataFrame (df2) and the matching rows from the left DataFrame (df1). If there is no match, the result will contain null for the columns from the left DataFrame.</p><p><strong>Syntax:</strong></p><pre>df1.join(df2, on=&quot;column_name&quot;, how=&quot;right&quot;)</pre><p><strong>Example:</strong></p><pre># All rows from df2, and matching rows from df1<br>right_joined_df = df1.join(df2, on=&quot;id&quot;, how=&quot;right&quot;)</pre><p><strong>Output:</strong></p><ul><li>All rows from df2 will be in the result. If there’s no match in df1, the corresponding columns will be null.</li></ul><h3>4. Full Outer Join (outer or full)</h3><p>A <strong>full outer join</strong> returns all rows from both DataFrames. If a row has no matching row in the other DataFrame, the result will contain null for the missing values.</p><p><strong>Syntax:</strong></p><pre>df1.join(df2, on=&quot;column_name&quot;, how=&quot;outer&quot;)</pre><p><strong>Example:</strong></p><pre># All rows from both df1 and df2, with nulls where there&#39;s no match<br>outer_joined_df = df1.join(df2, on=&quot;id&quot;, how=&quot;outer&quot;)</pre><p><strong>Output:</strong></p><ul><li>All rows from both df1 and df2. If there’s no match in one DataFrame, the corresponding columns will be null.</li></ul><h3>5. Left Anti Join (left_anti)</h3><p>A <strong>left anti join</strong> returns all rows from the left DataFrame (df1) that do not have a matching row in the right DataFrame (df2). This is useful for filtering out matching rows.</p><p><strong>Syntax:</strong></p><pre>df1.join(df2, on=&quot;column_name&quot;, how=&quot;left_anti&quot;)</pre><p><strong>Example:</strong></p><pre># Rows from df1 that do not have a matching &#39;id&#39; in df2<br>left_anti_joined_df = df1.join(df2, on=&quot;id&quot;, how=&quot;left_anti&quot;)</pre><p><strong>Output:</strong></p><ul><li>All rows from df1 that do not have a corresponding match in df2.</li></ul><h3>6. Left Semi Join (left_semi)</h3><p>A <strong>left semi join</strong> returns all rows from the left DataFrame (df1) where there is a match in the right DataFrame (df2). Unlike the <strong>inner join</strong>, it does not include columns from the right DataFrame.</p><p><strong>Syntax:</strong></p><pre>df1.join(df2, on=&quot;column_name&quot;, how=&quot;left_semi&quot;)</pre><p><strong>Example:</strong></p><pre># Rows from df1 that have a matching &#39;id&#39; in df2<br>left_semi_joined_df = df1.join(df2, on=&quot;id&quot;, how=&quot;left_semi&quot;)</pre><p><strong>Output:</strong></p><ul><li>Only rows from df1 that have a match in df2. No columns from df2 are included.</li></ul><h3>Summary of Join Types:</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1005/1*UsPpTIrPvdBhqY5r5zo_Hg.png" /></figure><h3>Interview Response:</h3><p>In PySpark, we can perform different types of joins:</p><ol><li><strong>Inner Join</strong>: Returns only matching rows from both DataFrames.</li><li><strong>Left Join</strong>: Returns all rows from the left DataFrame and matching rows from the right DataFrame.</li><li><strong>Right Join</strong>: Returns all rows from the right DataFrame and matching rows from the left DataFrame.</li><li><strong>Full Outer Join</strong>: Returns all rows from both DataFrames, with nulls where there is no match.</li><li><strong>Left Anti Join</strong>: Returns rows from the left DataFrame that do not have a match in the right DataFrame.</li><li><strong>Left Semi Join</strong>: Returns rows from the left DataFrame that have a match in the right DataFrame, but does not include columns from the right DataFrame.</li></ol><p><strong>8)How to rename columns in a PySpark DataFrame ?</strong></p><p>In PySpark, we can rename columns using the withColumnRenamed() function. It allows us to rename a column at a time.</p><p>df = df.withColumnRenamed(‘old_name’, ‘new_name’)</p><p><strong>9) How to handle missing or null values in a PySpark DataFrame ?</strong></p><p>To handle missing or null values in a PySpark DataFrame, you can use several functions provided by PySpark. The main functions for dealing with null or missing data are:</p><ol><li><strong>na.drop()</strong> – Removes rows with null values.</li><li><strong>na.fill()</strong> – Fills null values with a specified value.</li><li><strong>na.replace()</strong> – Replaces a value with another value in the DataFrame.</li><li><strong>fillna()</strong> – Similar to na.fill(), but for filling missing data.</li><li><strong>dropna()</strong> – Similar to na.drop(), but more flexible.</li></ol><h3>Example:</h3><h4>Step 1: Create a sample DataFrame with null values</h4><pre>from pyspark.sql import SparkSession<br>from pyspark.sql.functions import col</pre><pre># Initialize SparkSession<br>spark = SparkSession.builder \<br>    .appName(&quot;HandleNullValues&quot;) \<br>    .master(&quot;local[*]&quot;) \<br>    .getOrCreate()</pre><pre># Sample DataFrame with null values<br>data = [(&quot;Alice&quot;, 25), (&quot;Bob&quot;, None), (&quot;Charlie&quot;, 30), (None, 35)]<br>columns = [&quot;name&quot;, &quot;age&quot;]</pre><pre>df = spark.createDataFrame(data, columns)</pre><pre># Show the DataFrame<br>df.show()</pre><h4>Output:</h4><pre>name    age<br>Alice   25<br>Bob     null<br>Charlie 30<br>null    35</pre><h3>1. Remove Rows with Null Values (dropna):</h3><p>Use dropna() to remove rows that contain null values.</p><pre># Remove rows with any null values<br>df_cleaned = df.dropna()</pre><pre>df_cleaned.show()</pre><h4>Output:</h4><pre>name    age<br>Alice   25<br>Charlie 30</pre><h3>2. Fill Null Values with a Specific Value (fillna):</h3><p>Use fillna() to replace null values with a specific value, such as 0 for numerical columns or &quot;Unknown&quot; for string columns.</p><pre># Fill null values with a default value (0 for age)<br>df_filled = df.fillna({&quot;age&quot;: 0})</pre><pre>df_filled.show()</pre><h4>Output:</h4><pre>name    age<br>Alice   25<br>Bob     0<br>Charlie 30<br>null    35</pre><h3>3. Fill Null Values with Different Values for Different Columns (fillna with dictionary):</h3><p>You can also specify different fill values for different columns.</p><pre># Fill null values in &#39;age&#39; column with 0 and &#39;name&#39; column with &quot;Unknown&quot;<br>df_filled = df.fillna({&quot;age&quot;: 0, &quot;name&quot;: &quot;Unknown&quot;})</pre><pre>df_filled.show()</pre><h4>Output:</h4><pre>name    age<br>Alice   25<br>Bob     0<br>Charlie 30<br>Unknown 35</pre><h3>4. Remove Rows with Null Values in Specific Columns (dropna with subset):</h3><p>If you only want to drop rows with null values in specific columns, you can use the subset argument.</p><pre># Remove rows where &#39;age&#39; column is null<br>df_cleaned = df.dropna(subset=[&quot;age&quot;])</pre><pre>df_cleaned.show()</pre><h4>Output:</h4><pre>name    age<br>Alice   25<br>Charlie 30<br>null    35</pre><h3>5. Replace Specific Values (replace):</h3><p>You can also replace specific values (like replacing “None” or a certain string with a valid value).</p><pre># Replace &#39;None&#39; with &#39;Unknown&#39; in &#39;name&#39; column<br>df_replaced = df.replace(&quot;None&quot;, &quot;Unknown&quot;, subset=[&quot;name&quot;])</pre><pre>df_replaced.show()</pre><h4>Output:</h4><pre>name    age<br>Alice   25<br>Bob     null<br>Charlie 30<br>Unknown 35</pre><p>In PySpark, there are several ways to handle missing or null values in a DataFrame:</p><p><strong>dropna()</strong>: Removes rows with null values. You can remove rows with nulls in all columns or specific columns using the subset parameter.</p><p>Example:</p><p>df.dropna() # Drops rows with any null values</p><p><strong>fillna()</strong>: Fills null values with a specific value, like 0 or &quot;Unknown&quot;. You can use a dictionary to fill different columns with different values.</p><ul><li>df.fillna({&quot;age&quot;: 0, &quot;name&quot;: &quot;Unknown&quot;})</li></ul><p><strong>replace()</strong>: Replaces specific values with another value, useful when you need to replace a particular string or numeric value in the DataFrame.</p><p>Example:</p><p>df.replace(&quot;None&quot;, &quot;Unknown&quot;, subset=[&quot;name&quot;])</p><p><strong>10) Create a new column derived from existing columns in pyspark dataframe ?</strong></p><p>In PySpark, to create a new column derived from existing columns, we use the withColumn() function. This function allows us to perform operations on the existing columns to generate the new column. You can apply various transformations like arithmetic operations, string manipulations, or conditional statements.</p><p><strong>Example 1:</strong> To create a new column by adding two existing columns:</p><pre>df.withColumn(&quot;new_column&quot;, col(&quot;col1&quot;) + col(&quot;col2&quot;))</pre><p><strong>Example 2:</strong> To create a new column based on a condition:</p><pre>from pyspark.sql.functions import when<br>df.withColumn(&quot;new_column&quot;, when(col(&quot;age&quot;) &gt; 30, &quot;Old&quot;).otherwise(&quot;Young&quot;))</pre><p><strong>Example 3:</strong> To create a new column by concatenating two string columns:</p><pre>from pyspark.sql.functions import concat_ws<br>df.withColumn(&quot;full_name&quot;, concat_ws(&quot; &quot;, col(&quot;first_name&quot;), col(&quot;last_name&quot;)))</pre><p>This approach allows you to manipulate and derive new columns based on the data in your existing DataFrame.</p><p>#PySpark #InterviewPreparation #DataScience #Dataengineer #BigData #CareerAdvice</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9b7ee05d6b51" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>