<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Mithlesh Vishwakarma on Medium]]></title>
        <description><![CDATA[Stories by Mithlesh Vishwakarma on Medium]]></description>
        <link>https://medium.com/@mithsv20?source=rss-3ac234fbc79f------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*M96fTGPBr45fldG5x4wofg.jpeg</url>
            <title>Stories by Mithlesh Vishwakarma on Medium</title>
            <link>https://medium.com/@mithsv20?source=rss-3ac234fbc79f------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 08:23:08 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@mithsv20/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Spark vs Hive behavior over “collect_set”]]></title>
            <link>https://medium.com/globant/spark-vs-hive-behavior-over-collect-set-95f0ad31206f?source=rss-3ac234fbc79f------2</link>
            <guid isPermaLink="false">https://medium.com/p/95f0ad31206f</guid>
            <dc:creator><![CDATA[Mithlesh Vishwakarma]]></dc:creator>
            <pubDate>Thu, 20 Jul 2023 13:26:59 GMT</pubDate>
            <atom:updated>2023-07-20T13:26:59.057Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*8Cp_pTmE91JlMsKw.png" /></figure><h3>Introduction</h3><p>When working with big data in distributed environments like <a href="https://spark.apache.org/docs/latest/">Spark</a> and <a href="https://hive.apache.org/">Hive</a>, it is not uncommon to come across situations where the same query produces different results. This can be due to various factors, such as differences in the execution engines, optimisation techniques, or even the underlying data format.</p><p>One such scenario where the results differ is when using the collect_set function in Spark and Hive. In this blog post, we’ll take a deep dive into the differences between <a href="https://spark.apache.org/docs/3.1.1/api/python/reference/api/pyspark.sql.functions.collect_set.html">collect_set</a> in <a href="https://spark.apache.org/docs/latest/">Spark</a> and <a href="https://hive.apache.org/">Hive</a> and explore the reasons behind these differences.</p><h3>What is collect_set in Spark and Hive?</h3><p>Collect_set is a built-in function in Spark and Hive that returns a set of unique elements from a column. In Spark, collect_set is part of the DataFrame API and can be used with Spark SQL or PySpark. In Hive, collect_set is used with the SELECT statement and can be used with HiveQL.</p><h3>Why does collect_set produce different results in Spark and Hive?</h3><p>The main reason why collect_set produces different results in Spark and Hive is due to the order of elements. In Spark, the order of elements in a set is not guaranteed. This means that if two sets have the same elements, but in a different order, they will not be considered equal. In Hive, the order of elements in a set is guaranteed. This means that if two sets have the same elements but in a different order, they will still be considered equal.</p><h3>Use-Case</h3><p>Let’s consider a scenario where we have a table input_data containing information about user preferences. The table has columns user_id, domain_name, paid_or_free, r_num, and idx_period. We want to group the data by user_id and paid_or_free columns, collect the distinct domain_name values, and concatenate them using a colon separator.</p><pre>WITH input_data AS (<br>  SELECT &#39;00024&#39; AS user_id, &#39;TM&#39; AS domain_name, &#39;PAID&#39; AS paid_or_free, 2 AS r_num, &#39;2022-12&#39; AS idx_period<br>  UNION ALL<br>  SELECT &#39;00024&#39; AS user_id, &#39;ENT&#39; AS domain_name, &#39;PAID&#39; AS paid_or_free, 1 AS r_num, &#39;2022-12&#39; AS idx_period<br>)<br>SELECT<br>    user_id, paid_or_free, <br>    concat_ws(&#39;:&#39; , collect_set(domain_name)) as domain_switch<br>FROM input_data<br>GROUP BY<br>     user_id,<br>     paid_or_free</pre><p>The expected output of this query is:</p><pre>user_id                  | paid_or_free | domain_switch<br>-------------------------|--------------|--------------<br>00024                    | PAID         | TM:ENT</pre><p>In Hive, the output of this query matches the expected output. However, in Spark, the output is different:</p><pre>user_id                  | paid_or_free | domain_switch<br>-------------------------|--------------|--------------<br>00024                    | PAID         | ENT:TM</pre><p>As we can see, the order of the domain_name values in the domain_switch the column is reversed in Spark compared to Hive. This can happen because the collect_set function does not guarantee the order in which the elements are collected. In Hive, the default behavior is to preserve the order of the elements, whereas in Spark, the order is not guaranteed.</p><p>To ensure consistent behavior across both Spark and Hive, we can sort the elements before concatenating them using the sort_array function, as shown below:</p><pre>WITH input_data AS (<br>  SELECT &#39;00024&#39; AS user_id, &#39;TM&#39; AS domain_name, &#39;PAID&#39; AS paid_or_free, 2 AS r_num, &#39;2022-12&#39; AS idx_period<br>  UNION ALL<br>  SELECT &#39;00024&#39; AS user_id, &#39;ENT&#39; AS domain_name, &#39;PAID&#39; AS paid_or_free, 1 AS r_num, &#39;2022-12&#39; AS idx_period<br>)<br>SELECT<br>    user_id, paid_or_free, <br>    concat_ws(&#39;:&#39; , sort_array(collect_set(domain_name))) as domain_switch<br>FROM input_data<br>GROUP BY<br>     user_id,<br>     paid_or_free</pre><p>The above solution is applicable only when you are able to modify the query over both the environment Spark and Hive.</p><p>Here is one more proposed solution using which we can generate the expected result in Spark as Hive</p><pre>WITH input_data AS (<br>  SELECT &#39;00024&#39; AS user_id, &#39;TM&#39; AS domain_name, &#39;PAID&#39; AS paid_or_free, 2 AS r_num, &#39;2022-12&#39; AS idx_period<br>  UNION ALL<br>  SELECT &#39;00024&#39; AS user_id, &#39;ENT&#39; AS domain_name, &#39;PAID&#39; AS paid_or_free, 1 AS r_num, &#39;2022-12&#39; AS idx_period<br>)<br>SELECT user_id, paid_or_free, IF(first_domain_name is not null and second_domain_name is not null, concat(first_domain_name, &#39;:&#39;, second_domain_name), domain_name) as domain_switch <br>FROM (select concat_ws(&#39;:&#39; , collect_set(domain_name)) as domain_name, member_guid, idx_period, max(first_domain_name) as first_domain_name, max(second_domain_name) as second_domain_name <br>  FROM <br>  (select *, null as second_domain_name, domain_name as first_domain_name from input_datawhere r_num = 2 <br>  union all <br>  select *, domain_name as second_domain_name, null as first_domain_name from input_data where r_num = 1)A <br>  group by member_guid, idx_period)B</pre><p>This proposed solution is validated only when the number of elements in domain_name is two. Developers can enhance it further for more different domains.</p><pre>user_id                  | paid_or_free | domain_switch<br>-------------------------|--------------|--------------<br>00024                    | PAID         | TM:ENT</pre><h3>Possible cause of the differences</h3><p>One possible cause is differences in the underlying data structures used by Spark and Hive to store and manipulate data (e.g., Spark uses Resilient Distributed Datasets, or RDDs, while Hive uses the Hadoop Distributed File System or HDFS).</p><p>These differences may affect the way collect_set operates, particularly when dealing with complex data types or nested structures.</p><h3>Conclusion</h3><p>Collecting unique elements from a dataset is a common task in data processing, and the collect_set function is used for this purpose in both Spark and Hive. However, differences in the order of elements collect_set can produce different results in Spark and Hive. In this blog, we explored the reasons behind these differences and provided some approaches for handling them. As data analysts, it&#39;s important to be aware of these differences and choose the appropriate approach for our specific use case.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=95f0ad31206f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/globant/spark-vs-hive-behavior-over-collect-set-95f0ad31206f">Spark vs Hive behavior over “collect_set”</a> was originally published in <a href="https://medium.com/globant">Globant</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Multiple Sinks In Spark Structured Streaming]]></title>
            <link>https://medium.com/globant/multiple-sinks-in-spark-structured-streaming-38997d9a59e9?source=rss-3ac234fbc79f------2</link>
            <guid isPermaLink="false">https://medium.com/p/38997d9a59e9</guid>
            <dc:creator><![CDATA[Mithlesh Vishwakarma]]></dc:creator>
            <pubDate>Thu, 13 Jul 2023 13:26:17 GMT</pubDate>
            <atom:updated>2023-07-13T13:26:17.895Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/653/0*vaMIC6Uau1umG16f.png" /><figcaption>ref: <a href="https://qimia.io/en/blog/developing-streaming-applications-spark-structured-streaming/">https://qimia.io/en/blog/developing-streaming-applications-spark-structured-streaming/</a></figcaption></figure><p>While creating a data pipeline with near real-time execution, there is an interesting scenario that I have faced while reading sources, transforming complex data till writing data into multiple sinks/writes. While looking out for the solutions, I have gone through many articles explaining individual concepts of pipeline creation and execution; I believe this is the scenario with which most developers struggle with the constraints of streaming during real-time execution.</p><p>In this article, I will cover the behavior of <a href="https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html">Spark Structure Streaming</a> over single input and multiple outputs scenarios and go into the depth of its execution.</p><h3>Problem Statement</h3><p>While we read data from a single source to multiple sinks/writes, multiple jobs are getting created, which causes the same data to be read multiple times from the source.</p><h3>Spark Behavior when Splitting a Stream into multiple sinks</h3><p>To generate the possible scenario, we are consuming data from Kafka using Structured Streaming and writing the processed dataset to S3 while using multiple writers in a single job.</p><p>When writing a dataset created from a Kafka input source, as per basic understanding in the execution, we should have a single job containing single input with multiple outputs. Still, when I checked the spark-UI, multiple jobs were created for each output.</p><p>Since we were forking the output to multiple sinks, we had to provide the checkpointing for each sink/write. As we were forking the output, which caused the creation of multiple streaming queries(jobs).</p><p>Due to Spark parallelism, multiple streaming queries(jobs) can be scheduled independently and can maintain their own <a href="https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#recovering-from-failures-with-checkpointing">Checkpoint Location</a>; due to this behavior, the data is read from the input source independently (multiple times based on no sinks/writes), the same thing is reflected in Spark-UI.</p><p>In conclusion, the data is expected to be read multiple times from the input topic even when the same dataset is created from that source and write that data to multiple sinks/writes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*n88u_67fw8ZqPVNs.png" /></figure><p>In the above image, the timeline portion exactly shows the same response where each sink has its job ID, and it will repeatedly start reading data from the source as an individual.</p><p>The above screenshot gives a better idea of how Spark Structured Streaming internally works when executing multiple writers over a single read.</p><p>For the same batch, it creates multiple jobs with the same query plan, hence showing multiple read attempts while writing to multiple sinks.</p><p>If we have N sinks, it means you have N queries. Each query has its own checkpointing and fetches data from the source independently.</p><p>Each sink/write creates a new stream, runs independently, and offsets are picked from their own checkpoint location. Different group IDs are created for each stream as if you are running different read + write streams.</p><p><strong>Cons:</strong> This causes performance degradation when we deal with a massive dataset in our source/input. The above use case has tested over 51K messages in the Kafka topic, showing 4 seconds for individual write streams.</p><p>Below is the implementation of multiple sinks/writes.</p><h3>Solution: Achieving Writes to multiple sinks using forEachBatch writer</h3><p>We can prevent multiple read attempts while writing to multiple sinks using forEachBatch write.</p><p>During this execution, we are consuming data from Kafka using Structured Streaming and writing the processed data set to S3 while using multiple writers in a single job using forEachBatch write.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/451/0*P1nARqsM8TRoIhrE.png" /><figcaption>ref: <a href="https://jaceklaskowski.gitbooks.io/spark-structured-streaming/content/spark-sql-streaming-MicroBatchExecution.html">https://jaceklaskowski.gitbooks.io/spark-structured-streaming/content/spark-sql-streaming-MicroBatchExecution.html</a></figcaption></figure><p>The difference between normal data execution and with forEachBatch is during forEachBatch the single streaming query executes till forEachBatch process execution.</p><p>Inside forEachBatch we will get the batch <a href="https://www.databricks.com/glossary/what-are-dataframes">dataframe</a> behavior of the stream where we can write to multiple destinations.</p><p>Due to a single streaming query, we are maintaining a single Checkpoint Location.</p><p>With forEachBatch writer, we have the opportunity to cache the dataframe.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*B3oHkJR2NPbU6mGq.png" /></figure><p>The above image shows performance optimization over the multiple sinks/writes without reading from the source multiple times.</p><p>In the initial job, the data is read from the source, and using forEachBatch it helps to get cached and retrieve the data for the rest of the job.</p><p><strong>Cons:</strong> This only supports sequential execution, though we cause max computing consumption issues.</p><p>Below is the code implementation using forEachBatch writes:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d548c4c18e27c26a6ef71aed30ba1d42/href">https://medium.com/media/d548c4c18e27c26a6ef71aed30ba1d42/href</a></iframe><p>This code reads data from a Kafka stream and writes it to three different Parquet files. The code uses a streaming query to process the data in batches and calls a function to write using forEachBatchto write the Parquet files. The function forEachBatchto prevent reading multiple reads from the source. The code also uses a checkpoint to ensure the data is not lost if the Spark application fails.</p><h3>Conclusion</h3><p>In this article, we discussed the behavior of Spark Structured Streaming when writing data to multiple sinks. We saw that by default, Spark will create a separate streaming query for each sink, which can lead to performance degradation and multiple reads of the same data from the source. To avoid this, we can use the forEachBatch writer, which allows us to write to multiple sinks within a single streaming query. The forEachBatch writer also allows us to cache the data frame, which can further improve performance.</p><p>Here are some of the key takeaways from this article:</p><ul><li>Spark Structured Streaming will create a separate streaming query for each sink by default.</li><li>This can lead to performance degradation and multiple reads of the same data from the source.</li><li>To avoid this, we can use the forEachBatch writer.</li><li>The forEachBatch writer allows us to write to multiple sinks within a single streaming query.</li><li>It also allows us to cache the data frame, further improving performance.</li></ul><p>I hope you found this article helpful. If you have any questions, please feel free to comment below.</p><p>Feel free to follow me on <a href="https://www.linkedin.com/in/mithlesh-vishwakarma-25248813b/">LinkedIn</a> for more articles.</p><p>Here are some additional resources that you may find helpful:</p><ul><li>Spark Structured Streaming documentation: <a href="https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html">https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html</a></li><li>Spark Structured Streaming multiple sinks: <a href="https://stackoverflow.com/questions/45642904/spark-structured-streaming-multiple-sinks">https://stackoverflow.com/questions/45642904/spark-structured-streaming-multiple-sinks</a></li><li>Spark Structured Streaming forEachBatch writer: <a href="https://jaceklaskowski.gitbooks.io/spark-structured-streaming/content/spark-sql-streaming-MicroBatchExecution.html">https://jaceklaskowski.gitbooks.io/spark-structured-streaming/content/spark-sql-streaming-MicroBatchExecution.html</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=38997d9a59e9" width="1" height="1" alt=""><hr><p><a href="https://medium.com/globant/multiple-sinks-in-spark-structured-streaming-38997d9a59e9">Multiple Sinks In Spark Structured Streaming</a> was originally published in <a href="https://medium.com/globant">Globant</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Impact of Spark filter over Filtered View]]></title>
            <link>https://medium.com/globant/the-impact-of-spark-filter-over-filtered-view-c91e0e785002?source=rss-3ac234fbc79f------2</link>
            <guid isPermaLink="false">https://medium.com/p/c91e0e785002</guid>
            <dc:creator><![CDATA[Mithlesh Vishwakarma]]></dc:creator>
            <pubDate>Thu, 06 Jul 2023 12:23:52 GMT</pubDate>
            <atom:updated>2023-07-06T12:23:52.309Z</atom:updated>
            <content:encoded><![CDATA[<h3>The Impact of Spark Filter on Filtered Views</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*K7EJnMCaalAAD9J8ir9Vzg.png" /></figure><p><a href="https://spark.apache.org/docs/latest/">Apache Spark</a> can read data from a view that already has a filter applied during its creation. This raises questions about how Spark handles such scenarios. There are several possibilities, such as filtering on both the partition column and the view query or filtering only on the view query without considering the partition column. Let’s explore different scenarios and understand the implications of these executions.</p><h3>Setting up</h3><p>Let’s consider an example using a table called events that stores information about events. The events table has a column called event_date which represents the date of the event. Assume the data is partitioned based on the event_date column.</p><pre>CREATE TABLE events (<br>  event_id INT,<br>  event_date DATE,<br>  event_type VARCHAR(255),<br>  PRIMARY KEY (event_id)<br>)<br>PARTITION BY event_date;</pre><p>We create a view called current_month_events that contains data only for the current month using the following command:</p><pre>CREATE VIEW current_month_events AS<br>SELECT *<br>FROM events<br>WHERE event_date &gt;= CURRENT_DATE<br>AND event_date &lt; CURRENT_DATE + INTERVAL 1 MONTH;</pre><h3><strong>Use Case 1: Spark filter applied over the view partition column</strong></h3><p>If you read data from current_month_events and apply a filter on the event_date column, Spark will only read the data for the current month. This improves performance as Spark doesn’t need to read all the data from the events table.</p><p>For example, the following command reads data from current_month_events and filters it to return events that occurred on the 1st of the month:</p><pre>SELECT *<br>FROM current_month_events<br>WHERE event_date = CURRENT_DATE;</pre><p>This command only reads the data for the current month from the events table, leading to improved performance.</p><p>In conclusion, when you create a view of a table with a partitioned event_date column, you essentially create a new table with the same schema as the original table but containing only the data for the specified event_date partition. If you read data from the view using Spark and apply a filter on the event_date partition, Spark will only read the data from the specified partition, improving performance.</p><h3><strong>Use Case 2: View filter on the partition column, but Spark read filter on a different column</strong></h3><p>If the view filter is applied on the partition column, but the Spark read filter is not applied on the partition column, Spark will read the complete month of data and then apply the filter.</p><p>For instance, the view called current_month_events that only contains data for the current month. This view query filter is applied on the event_date partition column, ensuring that only data for the current month are available to Spark.</p><pre>SELECT *<br>FROM current_month_events<br>WHERE event_type = &#39;corporate&#39;;</pre><p>If you try to apply a filter to the view based on a different column than the partition column, Spark will read the complete month of data and apply the filter on top of the loaded partitions. However, this may not return any data if the filter doesn’t match.</p><h3><strong>Use Case 3: Filter on a day outside the range of the view query filter</strong></h3><p>If you apply a filter on a day that falls outside the range of the view query filter, Spark will not return any data. The view query filter defines the data available to Spark. If you try to filter the data to include days unavailable to Spark, no data will be returned.</p><p>For instance, the view called current_month_events that only contains data for the current month. This view query filter is applied to retrieve events that occurred two months ago from the current date:</p><pre>SELECT *<br>FROM current_month_events<br>WHERE event_date = CURRENT_DATE - INTERVAL 2 MONTH;</pre><p>This command will try to filter the data so that it includes events that happened two months ago. However, the current_month_events view only contains data for the current month. Therefore, Spark will not be able to return any data.</p><h3><strong>Use Case 4: Spark adds filter when applying a filter on a view that already has a filter</strong></h3><p>When Spark adds an additional filter to a view that already has a filter, the <a href="https://spark.apache.org/docs/3.0.0/web-ui.html#:~:text=An-,example,-of%20DAG%20visualization">DAG</a> of the query will have two stages. The first stage will read the data from the view, and the second stage will apply the additional filter.</p><p>The first stage of the <a href="https://spark.apache.org/docs/3.0.0/web-ui.html#:~:text=An-,example,-of%20DAG%20visualization">DAG</a> will be a simple scan of the view. This is because the view filter has already been applied to the data, so Spark does not need additional filters at this stage.</p><p>The second stage of the DAG will be a filter operation. This is because the Spark filter is not already satisfied by the view filter. The Spark filter will be applied to the data read from the view in the first stage.</p><p>The following is an example of the DAG of a query when Spark adds an additional filter when applying a filter on a view that already has a filter:</p><pre>Stage 1: Read Data From View<br>    - Read data from view `current_month_events`<br>    - Filter data by `event_date &gt;= CURRENT_DATE` and `event_date &lt; CURRENT_DATE + INTERVAL 1 MONTH`<br>Stage 2: Apply Spark Filter<br>    - Apply Spark filter `event_type = &#39;corporate&#39;` to the data read from the view in Stage 1<br>Result: Data that satisfies both the view filter and the Spark filter</pre><p>Note: It is important to note that the <a href="https://spark.apache.org/docs/3.0.0/web-ui.html#:~:text=An-,example,-of%20DAG%20visualization">DAG</a> of the query may vary depending on the specific Spark configuration and the data being processed.</p><h3>Conclusion</h3><ul><li>Spark can read data from a view that already has a filter applied during its creation.</li><li>There are different scenarios depending on how the Spark filter is applied.</li><li>In some cases, Spark will only read the data for the specified partition, improving performance.</li><li>In other cases, Spark will read the complete month of data and then apply the filter.</li><li>If you apply a filter on a day that falls outside the range of the view query filter, Spark will not return any data.</li><li>When Spark adds an additional filter to a view that already has a filter, the DAG of the query will have two stages.</li></ul><h3>Highlighting points</h3><ul><li>The performance implications of these different scenarios can vary depending on the size of the data set and the specific Spark configuration.</li><li>It is important to understand how Spark handles filters when working with views, in order to optimize performance and avoid unexpected results.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c91e0e785002" width="1" height="1" alt=""><hr><p><a href="https://medium.com/globant/the-impact-of-spark-filter-over-filtered-view-c91e0e785002">The Impact of Spark filter over Filtered View</a> was originally published in <a href="https://medium.com/globant">Globant</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Jupyter Notebook on EC2]]></title>
            <link>https://blog.devgenius.io/jupyter-notebook-on-ec2-c004bea5bb34?source=rss-3ac234fbc79f------2</link>
            <guid isPermaLink="false">https://medium.com/p/c004bea5bb34</guid>
            <category><![CDATA[ec2]]></category>
            <category><![CDATA[setup]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[spark]]></category>
            <category><![CDATA[jupyter-notebook]]></category>
            <dc:creator><![CDATA[Mithlesh Vishwakarma]]></dc:creator>
            <pubDate>Tue, 25 Apr 2023 12:18:08 GMT</pubDate>
            <atom:updated>2023-04-26T19:15:39.226Z</atom:updated>
            <content:encoded><![CDATA[<p>Jupyter Notebooks on EC2 came from needing more powerful resources for training a data processing and machine learning models. In my journey I have face a lot of challenges to setup Jupyter Notebook on EC2. Fortunately, there were some resources available online to guide me.</p><p>In this blog I am showing you the straightforward solution to achieve Jupyter Notebook connectivity on an Amazon EC2 instance. I’ll cover amazon linux operating system with easiest steps to setup Jupyter Notebook in your EC2 instance that allow Pyspark functionality to execute over it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/0*Rbw_t8vs27FjkLsQ.png" /></figure><h3>EC2 cluster creation basics:</h3><p>step 1:</p><p>Simply head on over to the <a href="https://aws.amazon.com/console/">AWS Console</a> and then head inside the EC2 Management Console:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ho4eDzG_C4AOUYzv.png" /></figure><p>step 2:</p><p>Launch a new instance by clicking the “launch Instance” button from the Instances console:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*AnFqB9_ck1TWv9R4.png" /></figure><p>step 3:</p><p>In the launch Wizard, provide a name to your EC2 instance, and then choose the AMI to “Amazon Linux”:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*HH98w3Lp2LsuulKW.png" /></figure><p>step 4:</p><p>Scroll down and choose the Instance types according to your requirements and also select a key pair by using an already existing one (Make sure it is available on your local pc) or by creating a new one:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*BM1HR9hxwBae2GPv.png" /></figure><p>step 5:</p><p>After that, click on the “Launch Instance” button at the bottom of the page:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*VOZd1R_W-zEMnfJP.png" /></figure><p>step 6:</p><p>Click on the name of the EC2 and then click on the security tab available below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pbjyBjxZi5RL5DyYftrCAg.png" /></figure><p>Click on available security groups it will redirect you to ec2 security group tab.</p><p>Inbound rules &gt; Edit inbound rules</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JiDwcirNZiRO3tJwZSP0Zw.png" /></figure><p>step 7:</p><p>Add new rule to open Jupyter Notebook port for the EC2 instance</p><p>Add rule &gt; Type {custom TCP} &gt; Port range {8888} &gt; Source {0.0.0.0/0}</p><p>Click on Save rules</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KHrYZVYT87VBsURrP2AKYw.png" /></figure><p>Step 8:</p><p>Come back to EC@ instance wizard</p><p>Click on the name of the EC2 and then click on the connect button:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*QRZac_7TBgLUdvMy.png" /></figure><p>step 9:</p><p>Copy the last command on the SSH client tab:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*jzCtw2IPQt2VaQ2o.png" /></figure><p>step11:</p><p>Before executing ssh connectivity command you need to ensure the permission status for downloaded .pem file. You can setup the proper permission with below command:</p><pre>chmod 600 LHKeyPair1.pem</pre><p>Replace the key pair name (“LHKeyPair1 for this post”) with the complete path of the key pair on your local machine, and then paste the command inside a Command Prompt and press “Enter”. If you are prompted about the connection, simply type “yes” and hit “Enter” again:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*FWNgEaoC-szwa4iD.png" /></figure><p>Wait for a couple of seconds, and you will be connected to the EC2 instance:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*uYv9FrB2pFGEK3h3.png" /></figure><h3>Setup Jupyter Notebook on Amazon Linux EC2 instance:</h3><p>step 1:</p><p>install or check python installation</p><pre>sudo yum update<br>sudo yum install python3<br>python3 -V</pre><p>step 2:</p><p>install or check pip installation</p><pre>sudo yum install python3-pip<br>pip3 -V</pre><p>step 3:</p><p>install Jupyter Notebook</p><pre>pip3 install jupyter</pre><p>awesome..!! now till this point you have setup Jupyter Notebook in your EC2 instance. Now you can launch notebook by executing below command:</p><pre>jupyter notebook --ip=0.0.0.0 --no-browser</pre><p>Output of above command give you result like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VBgcmeKvBAlu5ZwkBBDJBg.png" /></figure><p>Where you can copy the localhost:8088 url and paste it over the browser.</p><p>You need to replace the localhost or local ip(127.0.0.1) with instance DNS name(available over EC2 instance wizard).</p><p>Now you are able to access the Jupyter Notebook.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vrfXCZ_pf3qvwQXHsUJjqA.png" /></figure><p>You need few more setup process to perform pyspark over it.</p><p>step 4:</p><p>install pyspark through pip:</p><pre>pip3 install pyspark</pre><p>step5:</p><p>install and setup java environment variable:</p><pre>sudo yum install java-11*<br>java --version</pre><p>check installation directory:</p><pre>whereis java</pre><p>you’ll get output like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rNUNyTSSXrVOpX3FHBCD2g.png" /></figure><p>setup java environment variable:</p><pre>export JAVA_HOME=/usr/<br>echo $JAVA_HOME<br>source .bashrc</pre><p>Congrats..!! your setup is completely done for amazon linux EC2 instance. Now you can try it by executing below command:</p><pre>jupyter notebook --ip=0.0.0.0 --no-browser</pre><p>Keep Learning..!!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c004bea5bb34" width="1" height="1" alt=""><hr><p><a href="https://blog.devgenius.io/jupyter-notebook-on-ec2-c004bea5bb34">Jupyter Notebook on EC2</a> was originally published in <a href="https://blog.devgenius.io/">Dev Genius</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[When collect_set Produces Different Results in Spark and Hive]]></title>
            <link>https://blog.devgenius.io/when-collect-set-produces-different-results-in-spark-and-hive-a1514dacfe70?source=rss-3ac234fbc79f------2</link>
            <guid isPermaLink="false">https://medium.com/p/a1514dacfe70</guid>
            <category><![CDATA[pyspark]]></category>
            <category><![CDATA[spark]]></category>
            <category><![CDATA[data-engineer]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[hive]]></category>
            <dc:creator><![CDATA[Mithlesh Vishwakarma]]></dc:creator>
            <pubDate>Wed, 29 Mar 2023 06:39:36 GMT</pubDate>
            <atom:updated>2023-06-27T06:09:51.693Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction:</h3><p>When working with big data in distributed environments like Spark and Hive, it is not uncommon to come across situations where the same query produces different results. This can be due to various factors, such as differences in the execution engines, optimisation techniques, or even the underlying data format.</p><p>One such scenario where the results differ is when using the collect_set function in Spark and Hive. In this blog post, we’ll take a deep dive into the differences between collect_set in Spark and Hive and explore the reasons behind these differences.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BU_xLVtIALSz3_4af78gYQ.png" /></figure><p>What is collect_set in Spark and Hive?</p><p>Collect_set is a built-in function in Spark and Hive that returns a set of unique elements from a column. In Spark, collect_set is part of the DataFrame API and can be used with Spark SQL or PySpark. In Hive, collect_set is used with the SELECT statement and can be used with HiveQL.</p><p>Why does collect_set produce different results in Spark and Hive?</p><p>The main reason why collect_set produces different results in Spark and Hive is due to the order of elements. In Spark, the order of elements in a set is not guaranteed. This means that if two sets have the same elements but in a different order, they will not be considered equal. In Hive, the order of elements in a set is guaranteed. This means that if two sets have the same elements but in a different order, they will still be considered equal.</p><h3>Use-Case:</h3><p>Let’s consider a scenario where we have a table input_data containing information about user preferences. The table has columns user_id, domain_name, paid_or_free, r_num, and idx_period. We want to group the data by user_id and paid_or_free columns, collect the distinct domain_name values, and concatenate them using a colon separator.</p><pre>WITH input_data AS (<br>  SELECT &#39;000242A8628725360A495C85&#39; AS user_id, &#39;TM&#39; AS domain_name, &#39;PAID&#39; AS paid_or_free, 2 AS r_num, &#39;2022-12&#39; AS idx_period<br>  UNION ALL<br>  SELECT &#39;000242A8628725360A495C85&#39; AS user_id, &#39;ENT&#39; AS domain_name, &#39;PAID&#39; AS paid_or_free, 1 AS r_num, &#39;2022-12&#39; AS idx_period<br>)<br>SELECT<br>    user_id, paid_or_free, <br>    concat_ws(&#39;:&#39; , collect_set(domain_name)) as domain_switch<br>FROM input_data<br>GROUP BY<br>     user_id,<br>     paid_or_free</pre><p>The expected output of this query is:</p><pre>user_id                  | paid_or_free | domain_switch<br>-------------------------|--------------|--------------<br>000242A8628725360A495C85 | PAID         | TM:ENT</pre><p>In Hive, the output of this query matches the expected output. However, in Spark, the output is different:</p><pre>user_id                  | paid_or_free | domain_switch<br>-------------------------|--------------|--------------<br>000242A8628725360A495C85 | PAID         | ENT:TM</pre><p>As we can see, the order of the domain_name values in the domain_switch column is reversed in Spark compared to Hive. This can happen because the collect_set function does not guarantee the order in which the elements are collected. In Hive, the default behaviour is to preserve the order of the elements, whereas in Spark, the order is not guaranteed.</p><p>To ensure consistent behaviour across both Spark and Hive, we can sort the elements before concatenating them using the sort_array function, as shown below:</p><pre>WITH input_data AS (<br>  SELECT &#39;000242A8628725360A495C85&#39; AS user_id, &#39;TM&#39; AS domain_name, &#39;PAID&#39; AS paid_or_free, 2 AS r_num, &#39;2022-12&#39; AS idx_period<br>  UNION ALL<br>  SELECT &#39;000242A8628725360A495C85&#39; AS user_id, &#39;ENT&#39; AS domain_name, &#39;PAID&#39; AS paid_or_free, 1 AS r_num, &#39;2022-12&#39; AS idx_period<br>)<br>SELECT<br>    user_id, paid_or_free, <br>    concat_ws(&#39;:&#39; , sort_array(collect_set(domain_name))) as domain_switch<br>FROM input_data<br>GROUP BY<br>     user_id,<br>     paid_or_free</pre><p>Above solution is applicable only when you are able to modify query over both the environment Spark and Hive.</p><p>Here is one more proposed solution using which we can generate the expected result in Spark as Hive</p><pre>WITH input_data AS (<br>  SELECT &#39;000242A8628725360A495C85&#39; AS user_id, &#39;TM&#39; AS domain_name, &#39;PAID&#39; AS paid_or_free, 2 AS r_num, &#39;2022-12&#39; AS idx_period<br>  UNION ALL<br>  SELECT &#39;000242A8628725360A495C85&#39; AS user_id, &#39;ENT&#39; AS domain_name, &#39;PAID&#39; AS paid_or_free, 1 AS r_num, &#39;2022-12&#39; AS idx_period<br>)<br>SELECT user_id, paid_or_free, IF(first_domain_name is not null and second_domain_name is not null, concat(first_domain_name, &#39;:&#39;, second_domain_name), domain_name) as domain_switch <br>FROM (select concat_ws(&#39;:&#39; , collect_set(domain_name)) as domain_name, member_guid, idx_period, max(first_domain_name) as first_domain_name, max(second_domain_name) as second_domain_name <br>  FROM <br>  (select *, null as second_domain_name, domain_name as first_domain_name from input_datawhere r_num = 2 <br>  union all <br>  select *, domain_name as second_domain_name, null as first_domain_name from input_data where r_num = 1)A <br>  group by member_guid, idx_period)B</pre><p>This proposed solution is validate only when number of element in domain_name is two. Developer can enhance it further for more different domains.</p><pre>user_id                  | paid_or_free | domain_switch<br>-------------------------|--------------|--------------<br>000242A8628725360A495C85 | PAID         | TM:ENT</pre><h3>Possible causes of the differences</h3><ul><li>Mention that one possible cause is differences in the underlying data structures used by Spark and Hive to store and manipulate data (e.g., Spark uses Resilient Distributed Datasets, or RDDs, while Hive uses the Hadoop Distributed File System, or HDFS).</li><li>Explain that these differences may affect the way collect_set operates, particularly when dealing with complex data types or nested structures.</li></ul><h3>Conclusion</h3><p>Collecting unique elements from a dataset is a common task in data processing, and the collect_set function is used for this purpose in both Spark and Hive. However, due to differences in the order of elements, collect_set can produce different results in Spark and Hive. In this blog, we explored the reasons behind these differences and provided some approaches for handling them. As data analysts, it&#39;s important to be aware of these differences and choose the appropriate approach for our specific use case.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a1514dacfe70" width="1" height="1" alt=""><hr><p><a href="https://blog.devgenius.io/when-collect-set-produces-different-results-in-spark-and-hive-a1514dacfe70">When collect_set Produces Different Results in Spark and Hive</a> was originally published in <a href="https://blog.devgenius.io/">Dev Genius</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Getting Started — Setup Git CLI for Corporate Account]]></title>
            <link>https://blog.devgenius.io/getting-started-setup-git-cli-for-corporate-account-50fbdcafdfcc?source=rss-3ac234fbc79f------2</link>
            <guid isPermaLink="false">https://medium.com/p/50fbdcafdfcc</guid>
            <category><![CDATA[git]]></category>
            <category><![CDATA[github]]></category>
            <category><![CDATA[ssh]]></category>
            <category><![CDATA[ssh-keys]]></category>
            <category><![CDATA[setup]]></category>
            <dc:creator><![CDATA[Mithlesh Vishwakarma]]></dc:creator>
            <pubDate>Wed, 11 Jan 2023 15:21:18 GMT</pubDate>
            <atom:updated>2023-01-12T15:11:46.296Z</atom:updated>
            <content:encoded><![CDATA[<h3>Getting Started — Setup Git CLI for Corporate Account</h3><p>Before you start using Git, you have to make it available on your computer. Even if it’s already installed, it’s probably a good idea to update to the latest version. You can either install it as a package or via another installer, or download the source code and compile it yourself.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/900/0*OEBXC9Hb9EMKZmEI.png" /><figcaption>ref: <a href="https://www.hostinger.com/tutorials/basic-git-commands">https://www.hostinger.com/tutorials/basic-git-commands</a></figcaption></figure><h3>Installing on Linux</h3><p>If you’re on a Debian-based distribution, such as Ubuntu, try apt:</p><pre>$ sudo apt install git</pre><p>For more options, there are instructions for installing on several different Unix distributions on the Git website, at <a href="https://git-scm.com/download/linux">https://git-scm.com/download/linux</a>.</p><h3>Installing on macOS</h3><p>There are several ways to install Git on a Mac. The easiest is probably to install the Homebrew Tools. try brew:</p><pre>$ brew install git</pre><h3>Installing on Windows</h3><p>There are also a few ways to install Git on Windows. The most official build is available for download on the Git website. Just go to <a href="https://git-scm.com/downloads">latest version of Git</a> and choose appropriate version to install. Execute .exe file to install Git on windows.</p><h3>Verify Git Installation</h3><pre>$ git --version</pre><h3>Setup Git on CLI</h3><p>On corporate git project we are mostly trying cloning the project directly. Instead we need to setup ssh key to access it.</p><h4>Generate SSH Key</h4><pre>$ ssh-keygen -t rsa</pre><p>Generate ssh key with above “keygen” command after executing command you do not need to enter any credentials you just need to put blank enter/return.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/713/1*JrS89V33LLfxJWmaYZhm3w.jpeg" /></figure><h4>Copy SSH Key</h4><p>copy content of “id_rsa.pub” file available in .ssh folder of your home directory</p><pre>$ vi .ssh/id_rsa.pub</pre><h4>Go to Git Project</h4><p>Go to git project and select the repository you wanted to clone:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/774/1*XW5Pn5DAvOFLeHh7NEOB7Q.png" /></figure><p>If above option/link is not visible on “code” option here is one alternate option you can choose:</p><p>profile -&gt; your organisations SSH and GPG key -&gt; New SSH key</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QEuz0VBjX15-LL38FrosLg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fv-MH_0hP4j_34c4ISRcwA.png" /></figure><h4>Add new ssh key in account</h4><p>Click on add a new public key and you’ll get page to add new SSH key. Paste the copied “id_rsa.pub” content in Key section with title you want:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TkVTA6_UUKht5lFJ-Nk7Kw.png" /></figure><h4>Clone project on git CLI</h4><p>After completing the steps, you can clone the project using git url copied from clone ssh option:</p><pre>$ git clone git://git.kernel.org/pub/scm/git/git.git</pre><p>Thanks..!</p><p>references:</p><p><a href="https://git-scm.com/download/linux">https://git-scm.com/download/linux</a></p><p><a href="https://git-scm.com/book/en/v2/Getting-Started-Installing-Git">https://git-scm.com/book/en/v2/Getting-Started-Installing-Git</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=50fbdcafdfcc" width="1" height="1" alt=""><hr><p><a href="https://blog.devgenius.io/getting-started-setup-git-cli-for-corporate-account-50fbdcafdfcc">Getting Started — Setup Git CLI for Corporate Account</a> was originally published in <a href="https://blog.devgenius.io/">Dev Genius</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Spark Structured Streaming: Multiple Sinks/Writes]]></title>
            <link>https://blog.devgenius.io/spark-structured-streaming-multiple-sinks-writes-5dea139d4920?source=rss-3ac234fbc79f------2</link>
            <guid isPermaLink="false">https://medium.com/p/5dea139d4920</guid>
            <category><![CDATA[structured-streaming]]></category>
            <category><![CDATA[transformation]]></category>
            <category><![CDATA[checkpointing]]></category>
            <category><![CDATA[spark]]></category>
            <category><![CDATA[spark-streaming]]></category>
            <dc:creator><![CDATA[Mithlesh Vishwakarma]]></dc:creator>
            <pubDate>Tue, 15 Nov 2022 08:14:15 GMT</pubDate>
            <atom:updated>2022-11-15T16:52:15.009Z</atom:updated>
            <content:encoded><![CDATA[<h3>Spark Structured Streaming: Multiple Sinks</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/653/1*TcAv29jE_IRcuOHJa8Y_Kw.png" /><figcaption>ref: <a href="https://qimia.io/en/blog/developing-streaming-applications-spark-structured-streaming/">https://qimia.io/en/blog/developing-streaming-applications-spark-structured-streaming/</a></figcaption></figure><p>While creating data pipeline with near real-time execution there is a interesting scenario which I have faced while reading source, transforming complex data till writing data into multiple sinks/writes. While looking out for the solutions, I have gone through many articles explaining individual concept of pipeline creation and execution, I believe this is the scenario with which most of the developer struggle with the constraints of the streaming during real time execution.</p><p>In this article I am going to cover the behavior of spark structure streaming over single input and multiple output scenario and will go into the depth of its execution.</p><h3>Problem Statement</h3><p>While we read data from single source to multiple sinks/writes, multiple jobs are getting created, which causes same data to be read multiple times from the source.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/810/0*V1fswmaJQFtxP4ah" /><figcaption>ref: <a href="https://www.scnsoft.com/services/big-data/spark">https://www.scnsoft.com/services/big-data/spark</a></figcaption></figure><h3>Spark Behavior: When Splitting Stream into multiple sinks</h3><p>To generate the possible scenario we are consuming data from Kafka using structured streaming and writing the processed dataset to s3 while using multiple writer in a single job.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/471/0*RfurzUbDmj26xi7c.png" /><figcaption>Ref: <a href="https://jaceklaskowski.gitbooks.io/spark-structured-streaming/content/spark-sql-streaming-MicroBatchExecution.html">https://jaceklaskowski.gitbooks.io/spark-structured-streaming/content/spark-sql-streaming-MicroBatchExecution.html</a></figcaption></figure><p>When writing a dataset created from a Kafka input source, as per basic understanding in the execution we should have single job contains single input with multiple output, but when I have checked the spark-UI there were multiple job created for each output.</p><p>Since we were forking the output to multiple sinks, we have to provide the checkpointing for each sink/write. As we were forking the output which causes the creation of multiple streaming queries(jobs).</p><p>Due to spark parallelism these jobs can be scheduled independently and can maintain its own checkpoint location, due to this behavior the data is read from the input source independently(multiple-times based on no of sinks/writes), the same thing is reflected in spark-UI.</p><p>In conclusion it is expected that the data will be read multiple times from the input topic even when same dataset created from that source and write that data to multiple sinks/writes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-mv-5fMgIum6sA1Z1spbAQ.png" /><figcaption>Here is the observation screenshots to give better idea how spark structured streaming internally work when executing the multiple writer over single read.</figcaption></figure><p>In above image the timeline portion exactly showing the same response where each sink has its own job ID and it is going to start reading data from the source repeatedly as individual.</p><p>For the same batch It is creating multiple jobs with same query plan, hence showing multiple read attempt happened while write to multiple sinks.</p><p>If we have N sinks, it means you have N queries. Each query has its own checkpointing and fetches data from source independently.</p><p>Each sink/write creates new stream, they run independently, and offsets are picked from their own checkpoint location. Different group ID is created for each stream, it is as if you are running different read + write stream</p><p><strong>Cons:</strong> This causes performance degradation when we dealing with huge dataset in our source/input. Above use case has tested over 51K messages in Kafka topic, showing 4 seconds taking for individual write stream.</p><p>Below is the implementation of multiple sink/write.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/162e97f609704e88a97e86816670b7e5/href">https://medium.com/media/162e97f609704e88a97e86816670b7e5/href</a></iframe><h3>Solution: Achieving Writes to multiple sinks using forEachBatch writer</h3><p>The above scenario can be handled using forEachBatch writer.</p><p>During this execution we are consuming data from Kafka using structured streaming and writing the processed data set to s3 while using multiple writer in a single job using forEachBatch writer.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/451/0*7AyKA6r8sMoRj5B5.png" /><figcaption>ref: <a href="https://jaceklaskowski.gitbooks.io/spark-structured-streaming/content/spark-sql-streaming-MicroBatchExecution.html">https://jaceklaskowski.gitbooks.io/spark-structured-streaming/content/spark-sql-streaming-MicroBatchExecution.html</a></figcaption></figure><p>The difference between normal data execution or with forEachBatch is during forEachBatch single streaming query execute till forEachBatch process execution.</p><p>Inside forEachBatch we will get batch dataframe behavior of the stream where we can write to multiple destinations.</p><p>Due to single streaming query we are maintaining single checkpoint location.</p><p>With forEachBatch writer we have opportunity to cache the dataframe.</p><p>Below are the performance observations we got after running against same no. of records(51K):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9S4GaBrpri79PnpoA34IIw.png" /></figure><p><strong>Cons:</strong> This only support sequential execution. though we causes max computing consumption issue.</p><p>Below is the code implementation using forEachBatch write.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8b8bcaee4b91ec113384ef7dc0ff3508/href">https://medium.com/media/8b8bcaee4b91ec113384ef7dc0ff3508/href</a></iframe><p>In the spark-UI we are seeing sequential execution which can be made parallel using multi-threading…….. but that will be another article for another day.</p><p>Thanks for reading.</p><p>Feel free to follow me on <a href="https://www.linkedin.com/in/mithlesh-vishwakarma-25248813b/">LinkedIn</a> for more articles.</p><p>References: <a href="https://stackoverflow.com/questions/45642904/spark-structured-streaming-multiple-sinks?rq=1">https://stackoverflow.com/questions/45642904/spark-structured-streaming-multiple-sinks?rq=1</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5dea139d4920" width="1" height="1" alt=""><hr><p><a href="https://blog.devgenius.io/spark-structured-streaming-multiple-sinks-writes-5dea139d4920">Spark Structured Streaming: Multiple Sinks/Writes</a> was originally published in <a href="https://blog.devgenius.io/">Dev Genius</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>