<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Gaurav Patil on Medium]]></title>
        <description><![CDATA[Stories by Gaurav Patil on Medium]]></description>
        <link>https://medium.com/@gauravpatil_95?source=rss-e04a068ef4ab------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*hC6WKqbIzw0am6kc69SKiw.jpeg</url>
            <title>Stories by Gaurav Patil on Medium</title>
            <link>https://medium.com/@gauravpatil_95?source=rss-e04a068ef4ab------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 08:02:43 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@gauravpatil_95/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Seamless Data Processing: Spark Structured Streaming for AWS Kinesis Message Streams]]></title>
            <link>https://medium.com/globant/seamless-data-processing-spark-structured-streaming-for-aws-kinesis-message-streams-bfcc59102066?source=rss-e04a068ef4ab------2</link>
            <guid isPermaLink="false">https://medium.com/p/bfcc59102066</guid>
            <category><![CDATA[aws-kinesis]]></category>
            <category><![CDATA[apache-spark-streaming]]></category>
            <category><![CDATA[etl-pipeline]]></category>
            <category><![CDATA[data-processing]]></category>
            <category><![CDATA[kinesis-data-stream]]></category>
            <dc:creator><![CDATA[Gaurav Patil]]></dc:creator>
            <pubDate>Thu, 22 Jun 2023 13:11:46 GMT</pubDate>
            <atom:updated>2023-06-23T09:03:35.891Z</atom:updated>
            <content:encoded><![CDATA[<h4>Harnessing Spark Structured Streaming for efficient AWS Kinesis message processing and analysis</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*HISh7GXpyWcjE8cQ" /><figcaption>Photo by <a href="https://unsplash.com/@markusspiske?utm_source=medium&amp;utm_medium=referral">Markus Spiske</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Organizations increasingly rely on real-time insights to make critical business decisions in today&#39;s data-driven world. <a href="https://aws.amazon.com/streaming-data/">Streaming data</a> has become a valuable resource, providing a continuous flow of information that can be processed and analyzed in near real-time. <a href="https://aws.amazon.com/kinesis/">AWS Kinesis</a>, a fully managed streaming service, offers a powerful platform for capturing, processing, and storing streaming data. However, efficiently processing and analyzing this massive influx of data can be challenging. <a href="https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html">Spark Structured Streaming</a> is a high-level <a href="https://www.ibm.com/topics/api">API</a> built on <a href="https://spark.apache.org/">Apache Spark</a> that simplifies the development of scalable, fault-tolerant, and real-time data processing applications. By seamlessly integrating with AWS Kinesis, Spark Structured Streaming provides an ideal framework for unlocking the full potential of streaming data.</p><p>In this blog post, we will learn how to implement streaming data processing using AWS Kinesis and Spark Structured Streaming. We will explore the benefits of using Spark Structured Streaming for handling AWS Kinesis messages. Whether you are a data engineer, data scientist, or technology enthusiast looking to enhance your real-time data processing capabilities, this blog post will guide you on processing Kinesis stream messages using Spark Structured Streaming.</p><h3>Understanding AWS Kinesis</h3><p><a href="https://aws.amazon.com/kinesis/"><strong>AWS Kinesis</strong></a> is a fully managed streaming service offered by Amazon Web Services (AWS) that enables organizations to capture, process, and analyze real-time streaming data at scale. It provides a reliable and scalable platform to ingest and store massive volumes of data from various sources, such as websites, mobile applications, <a href="https://en.wikipedia.org/wiki/Internet_of_things">IoT</a> devices, and more. To comprehend the power of Spark Structured Streaming for AWS Kinesis message processing, it’s essential to understand the key components of AWS Kinesis.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*-a56GVEd-pQgA5CC.png" /><figcaption>AWS Kinesis Service Key Components</figcaption></figure><p>Let’s understand the key components of AWS Kinesis one by one.</p><ul><li><strong>Kinesis Data Streams:</strong> This component acts as the foundation of AWS Kinesis. It allows you to create and manage highly scalable and durable data streams. A stream comprises one or more shards, each providing a fixed-capacity unit. Data records are distributed across these shards, ensuring parallel processing and fault tolerance.</li><li><strong>Kinesis Data Firehose:</strong> This service simplifies loading streaming data into AWS data stores or analytics services. It automatically scales to match the data ingestion rate and handles data delivery, compression, and transformation. Kinesis Data Firehose can directly integrate with <a href="https://cloud.google.com/learn/what-is-a-data-lake">data lakes</a>, <a href="https://aws.amazon.com/redshift/">Amazon Redshift</a>, <a href="https://aws.amazon.com/s3/">Amazon S3</a>, and other AWS services for further processing and analysis.</li><li><strong>Kinesis Data Analytics:</strong> This component allows you to run real-time SQL queries on streaming data. With Kinesis Data Analytics, you can perform transformations, aggregations, filtering, and join operations on the incoming data streams. It provides a powerful and intuitive interface for processing and gaining insights from streaming data without the need for complex infrastructure management.</li></ul><h3>Understanding Spark Structured Streaming</h3><p><a href="https://spark.apache.org/"><strong>Apache Spark</strong></a> is a widely adopted open-source distributed computing framework renowned for its speed, scalability, and ease of use. Spark Structured Streaming is a high-level API built on top of Apache Spark that simplifies the development of real-time data processing applications. It provides a declarative and unified programming model that treats streaming data as continuous tables, enabling developers to apply the same set of operations for both batch and streaming data. Let’s understand how structured streaming works in Spark.</p><p>Spark Structured Streaming follows a <a href="https://hazelcast.com/glossary/micro-batch-processing/">micro-batch processing</a> model, dividing streaming data into small batches. It leverages the underlying Spark engine’s capabilities to process these micro-batches in parallel and provide low-latency and fault-tolerant processing. The architecture includes a streaming source, transformations and operations, and output sinks.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/803/1*YjMkTYcvJJzTbkBOT6VAnQ.png" /><figcaption>The programming model of Spark Structured Streaming</figcaption></figure><p>Spark Structured Streaming offers several benefits for processing and analyzing streaming data:</p><ul><li><strong>Unified Batch and Streaming Processing:</strong> Spark Structured Streaming provides a unified programming model seamlessly integrating batch and streaming data processing. It allows you to use the same <a href="https://spark.apache.org/docs/latest/sql-programming-guide.html">DataFrame</a> and <a href="https://spark.apache.org/docs/latest/sql-programming-guide.html">SQL API</a> for static (batch) and dynamic (streaming) data, simplifying the development and maintenance of data pipelines.</li><li><strong>High-Level API:</strong> The high-level API of Spark Structured Streaming abstracts away the complexities of handling streaming data. It offers intuitive constructs like DataFrames, <a href="https://spark.apache.org/docs/latest/sql-programming-guide.html">Datasets</a>, and SQL queries, enabling developers to express their data processing logic in a declarative and SQL-like manner.</li><li><strong>Fault Tolerance and Reliability:</strong> Spark Structured Streaming provides fault tolerance and reliability guarantees out of the box. It maintains end-to-end reliability by checkpointing metadata and saving intermediate results. If failures occur, it can recover and resume processing from the last known consistent state automatically.</li><li><strong>Scalability and Parallelism:</strong> Spark’s distributed processing engine allows you to scale your streaming applications horizontally across a cluster of machines. It automatically parallelizes the processing of data across multiple nodes, enabling high throughput and low-latency processing of large-scale streaming data.</li><li><strong>Stateful Stream Processing:</strong> Spark Structured Streaming supports stateful stream processing, which allows you to maintain and update the state as new data arrives. This is useful for scenarios that require aggregations, event-time processing, windowed computations, or maintaining session-based information.</li><li><strong>Rich Ecosystem and Integrations:</strong> Spark has a vast ecosystem and supports a wide range of data sources and <a href="https://www.macrometa.com/articles/what-are-data-sinks">data sinks</a>, making integrating with various systems and technologies easily. You can seamlessly connect to popular data storage systems, message queues, databases, and more, enabling flexible data ingestion and output options.</li><li><strong>Extensibility and Compatibility:</strong> Spark Structured Streaming is built on top of the Spark engine, which provides extensive libraries, APIs, and integrations for data processing, <a href="https://en.wikipedia.org/wiki/Machine_learning">machine learning</a>, <a href="https://medium.com/swlh/what-is-graph-analytics-9223d71c26d8">graph analytics</a>, and more. It is compatible with various programming languages, including <a href="https://www.scala-lang.org/">Scala</a>, <a href="https://www.java.com/">Java</a>, <a href="https://www.python.org/">Python</a>, and <a href="https://www.w3schools.com/r/">R</a>, allowing you to leverage your preferred language and existing codebase.</li></ul><p>These benefits make Spark Structured Streaming a powerful framework for building real-time data processing applications, enabling developers to handle complex streaming use cases with ease and scalability.</p><p>By combining the capabilities of Spark Structured Streaming and AWS Kinesis, organizations can leverage the strengths of both platforms to process, analyze, and gain real-time insights from their streaming data at scale.</p><h3>A hands-on example of processing Kinesis messages with Spark Structured Streaming</h3><p>In this section, you will see how we can use Spark Structured Streaming for processing Kinesis messages. For this, we will set up a new Kinesis stream. We will put some messages to a Kinesis stream. Then we will read messages using Spark Streaming and print messages on the console.</p><p>To create a Kinesis Data Stream, follow the steps below.</p><ol><li>Sign in to the AWS Management Console and open the Kinesis console at <a href="https://console.aws.amazon.com/kinesis">https://console.aws.amazon.com/kinesis</a>.</li><li>Choose <em>Data Streams</em> in the navigation pane.</li><li>In the navigation bar, expand the <em>Region</em> selector and choose a <em>Region</em>.</li><li>Choose <em>Create Kinesis stream</em>.</li><li>Enter a name for your stream (for example, orders-data-stream).</li><li>Enter <em>1</em> for the number of shards, but leave <em>Estimate the number of shards you&#39;ll need</em><strong> </strong>as is.</li><li>Choose <em>Create Kinesis stream</em>.</li></ol><p>The next step is to put some data in Kinesis Data Stream, which we created in the earlier step. We will need the <a href="https://pypi.org/project/boto3/">boto3</a> library to connect to the AWS Kinesis service, the <a href="https://pypi.org/project/fastavro/">FastAvro</a> library to compress data to the Avro file and the <a href="https://pypi.org/project/pyspark/">PySpark</a> library to work with Spark. Perform all necessary imports in your Python code as given in below code snippet.</p><pre>import boto3<br>import fastavro<br>import tempfile<br>from pyspark.sql import SparkSession<br>from pyspark.sql.functions import col <br>from pyspark.sql.avro.functions import from_avro</pre><p>Define variables for stream name and Avro message schema. In the below code snippet, you need to provide the name of your Kinesis stream in the kinesis_stream_name variable. Provide schema of the message in avro_schema variable for writing the message in Avro format.</p><pre># Kinesis data stream name<br>kinesis_stream_name = &#39;your-kinesis-stream-name&#39;<br><br># Avro message schema<br>avro_schema = {<br> &#39;type&#39;: &#39;record&#39;,<br> &#39;name&#39;: &#39;Message&#39;,<br> &#39;fields&#39;: [<br> {&#39;name&#39;: &#39;id&#39;, &#39;type&#39;: &#39;string&#39;},<br> {&#39;name&#39;: &#39;title&#39;, &#39;type&#39;: &#39;string&#39;}<br> ]<br>}</pre><p>Now we will create a new function generate_avro_record that takes two parameters like Avro schema and dictionary object to be put as Avro data. We will call schemaless_writer function of the FastAvro library to generate bytes format data.</p><pre>def generate_avro_record(avro_schema, json_data):<br>    avro_schema = fastavro.parse_schema(avro_schema)<br>    fp = tempfile.TemporaryFile()<br>    fastavro.schemaless_writer(fp, avro_schema, json_data)<br>    fp.seek(0)<br>    data = fp.read()<br>    return data</pre><p>Next, we will initialize Kinesis Client, and using this object, we will get serialized data using generate_avro_record function and put it into the Kinesis data stream.</p><pre># Initialize Kinesis client<br>kinesis_client = boto3.client(&#39;kinesis&#39;, region_name=&#39;your-aws-region&#39;)<br><br>data = [<br>    {&#39;id&#39;: &#39;1&#39;, &#39;title&#39;: &#39;Data Engineer&#39;},<br>    {&#39;id&#39;: &#39;2&#39;, &#39;title&#39;: &#39;Data Scientist&#39;},<br>    {&#39;id&#39;: &#39;3&#39;, &#39;title&#39;: &#39;System Admin&#39;},<br>    {&#39;id&#39;: &#39;4&#39;, &#39;title&#39;: &#39;BI Developer&#39;}<br>]<br><br># Put data into Kinesis stream<br>for json_data in data:<br>    avro_bytes = generate_avro_record(avro_schema, json_data)<br>    response = kinesis_client.put_record(<br>        StreamName=kinesis_stream_name,<br>        Data=avro_bytes,<br>        PartitionKey=&#39;1&#39;<br>    )<br>    if response[&#39;ResponseMetadata&#39;][&#39;HTTPStatusCode&#39;] == 200:<br>        print(&quot;Data was successfully put into Kinesis stream.&quot;)<br>    else:<br>        print(&quot;Failed to put data into Kinesis stream.&quot;)</pre><p>We will initialize a Spark Session for processing the streaming data, which connects to Kinesis and reads streaming data in the Spark program.</p><pre># Configure Spark Environment<br>spark = SparkSession.builder \<br>    .appName(&#39;KinesisSparkStreamingExample&#39;) \<br>    .getOrCreate()<br><br># Read Avro Data from Kinesis Stream in PySpark<br>streaming_df = spark.readStream \<br>    .format(&#39;kinesis&#39;) \<br>    .option(&#39;streamName&#39;, kinesis_stream_name) \<br>    .option(&#39;region&#39;, &#39;your-aws-region&#39;) \<br>    .option(&#39;awsAccessKey&#39;, &#39;your-aws-access-key&#39;) \<br>    .option(&#39;awsSecretKey&#39;, &#39;your-aws-secret-key&#39;) \<br>    .load()</pre><p>You can apply some transformations to your data according to your needs. For example, removing special characters from a string, tokenizing, concatenating two or more columns, etc. In the below code snippet, I have used from_avro function to deserialize Avro data stored as a Kinesis message using avro_schema which I have defined earlier.</p><pre>processed_df = streaming_df.select(from_avro(&#39;data&#39;, avro_schema).alias(&#39;avroData&#39;)) \<br>                          .select(&quot;avroData.id&quot;, &quot;avroData.title&quot;)</pre><p>You can write processed data to a file, <a href="https://developer.confluent.io/learn-kafka/apache-kafka/topics/">Kafka topic</a>, database, or output console. In below code snippet, I have used console as an output format.</p><pre># Define Output Sink<br>output_query = processed_df.writeStream \<br>    .format(&quot;console&quot;) \<br>    .outputMode(&quot;append&quot;) \<br>    .start()<br><br># Start the Streaming Query<br>output_query.awaitTermination()</pre><p>This way, we can use Spark to process messages in the Kinesis data stream.</p><p>When you want to apply a custom action or operation on each micro-batch of data in a streaming DataFrame, you can use the foreachBatch() function in Spark Structured Streaming. It provides a flexible way to process the data and perform actions not directly supported by built-in streaming operations or sinks.</p><p>The below example demonstrates the use of foreachBatch() function. In this code snippet, I have provided the Kinesis stream name in kinesis_stream_name variable. The process_micro_batch function is defined to handle each micro-batch of data. You can write your custom transformation logic inside this function. Each micro-batch represents a portion of the streaming data that has arrived within a specific time interval. The batch_id argument is the default argument passed to process_micro_batch function. It holds an integer-type value and can be utilized for processing or referencing purposes within the function. Generated batch_id is specific to each micro-batch and is not a system-wide or universally unique number. The foreachBatch() function takes a user-defined function as an argument, which is applied to each micro-batch of data.</p><pre>from pyspark.sql import SparkSession<br>from pyspark.sql.functions import col, current_timestamp<br><br># Define Kinesis stream name <br>kinesis_stream_name = &#39;micro-batch-test-stream&#39;<br><br># configure Spark Environment<br>spark = SparkSession.builder \<br>    .appName(&quot;CustomMicroBatchExample&quot;) \<br>    .getOrCreate()<br><br># Read data<br>streaming_df = spark.readStream \<br>    .format(&#39;kinesis&#39;) \<br>    .option(&#39;streamName&#39;, kinesis_stream_name) \<br>    .option(&#39;region&#39;, &#39;your-aws-region&#39;) \<br>    .option(&#39;awsAccessKey&#39;, &#39;your-aws-access-key&#39;) \<br>    .option(&#39;awsSecretKey&#39;, &#39;your-aws-secret-key&#39;) \<br>    .load()<br><br>streaming_df.writeStream \<br>    .foreachBatch(process_micro_batch) \<br>    .start() \<br>    .awaitTermination()</pre><h3>Summary</h3><p>Spark Structured Streaming, in combination with AWS Kinesis, offers a powerful solution for processing and analyzing streaming data. By integrating Spark with Kinesis, organizations can leverage the scalability and reliability of Kinesis for data ingestion while utilizing Spark’s advanced processing capabilities. The combination of Spark Structured Streaming and AWS Kinesis empowers organizations to build real-time data processing applications with ease, leveraging the scalability, durability, and reliability of Kinesis and the advanced processing capabilities of Spark. This integration enables businesses to extract valuable insights and make data-driven decisions from streaming data sources seamlessly and efficiently.</p><h3>References</h3><p><a href="https://aws.amazon.com/kinesis/data-streams/getting-started/">Getting started with AWS Kinesis Service</a></p><p><a href="https://www.macrometa.com/event-stream-processing/spark-structured-streaming">Overview of Spark Structured Streaming</a></p><p>Thank you!!!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bfcc59102066" width="1" height="1" alt=""><hr><p><a href="https://medium.com/globant/seamless-data-processing-spark-structured-streaming-for-aws-kinesis-message-streams-bfcc59102066">Seamless Data Processing: Spark Structured Streaming for AWS Kinesis Message Streams</a> was originally published in <a href="https://medium.com/globant">Globant</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How  to solve the “large number of small files” problem in Spark]]></title>
            <link>https://medium.com/globant/how-to-solve-a-large-number-of-small-files-problem-in-spark-21f819eb36d3?source=rss-e04a068ef4ab------2</link>
            <guid isPermaLink="false">https://medium.com/p/21f819eb36d3</guid>
            <category><![CDATA[hadoop]]></category>
            <category><![CDATA[pyspark]]></category>
            <category><![CDATA[spark-optimization]]></category>
            <category><![CDATA[spark]]></category>
            <category><![CDATA[big-data]]></category>
            <dc:creator><![CDATA[Gaurav Patil]]></dc:creator>
            <pubDate>Fri, 25 Nov 2022 13:51:36 GMT</pubDate>
            <atom:updated>2022-11-25T14:40:21.131Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>How to solve the “large number of small files” problem in Spark</strong></h3><h4>A solution for the &quot;large number of small files&quot; problem</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*0z9tCV33G377k8wd" /><figcaption>Photo by <a href="https://unsplash.com/@timwildsmith?utm_source=medium&amp;utm_medium=referral">Tim Wildsmith</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p><a href="https://en.wikipedia.org/wiki/Big_data"><strong>Big Data</strong></a> is no longer a buzzword because everyone knows the power of data and what it can do. <a href="https://spark.apache.org/"><strong>Spark</strong></a><strong> </strong>is a robust framework for processing Big Data. It has many advantages over traditional data processing frameworks but has some limitations. One of the most significant limitations is that it stores the output in many small-size files while using object storage systems like HDFS, AWS S3, etc. This is because Spark is a parallel processing framework, and when the parallel tasks process the data, they store the output in multiple files across the partitions. So let&#39;s say your Spark application has 200 parallel tasks<em> </em>and 10 partitions; then the Spark application can potentially store results into <em>200 * 10 = 2000 </em>files. Running this job multiple times daily will create even more files in object storage. This is called <strong>a large number of small files</strong> problem in Spark.</p><p>In this article, I shall explain different ways to solve this problem. This article will help Data Engineers optimize their Spark applications&#39; output storage.</p><h3>Why do we consider <strong>&quot;a large number of small files&quot;</strong> a problem?</h3><p>Below are some reasons why we consider &quot;a large number of small files&quot; as an issue.</p><ul><li><a href="https://hadoop.apache.org/docs/r1.2.1/hdfs_design.html#NameNode+and+DataNodes"><strong>Name Node</strong></a><strong> overhead</strong><br>In <a href="https://hadoop.apache.org/docs/r1.2.1/hdfs_design.html">HDFS</a>, files are split into blocks that are stored and replicated on the <a href="https://hadoop.apache.org/docs/r1.2.1/hdfs_design.html#NameNode+and+DataNodes">Data Nodes</a>. Each block has a size of 128 MB, and every block has metadata associated with it. The HDFS namespace tree and associated metadata are maintained as objects in the NameNode&#39;s Main Memory, each occupying approximately 150 bytes. So, the Name Node has to manage these objects and serve the read/write requests. A large number of small blocks increases the overhead of the Name Node. When the Spark application reads data from HDFS, It has to serve more read requests for blocks. See the figure below, which illustrates HDFS architecture well.</li></ul><figure><img alt="HDFS Architecture" src="https://cdn-images-1.medium.com/max/1024/1*dD1nikt4r0RbiKNdMHf2Zg.png" /><figcaption>HDFS Architecture</figcaption></figure><ul><li><strong>Increased metadata overhead on query execution</strong><br>Writing small files to object storage is very straightforward, but queries over them will run much slower or fail to complete. Querying many small files incurs overhead to read metadata, conduct a non-contiguous disk seek, open the file, close the file and repeat. The overhead is only a few milliseconds per file, but those milliseconds add up when you&#39;re querying millions of files. <a href="https://hive.apache.org/">Hive</a> is an excellent tool for querying files stored in <a href="https://hadoop.apache.org/docs/r1.2.1/hdfs_design.html">HDFS</a>. It is a data warehousing solution built on top of HDFS. It supports schema-on-write, partitioning, and indexing data to speed up queries, but none works when we have large numbers of small files in HDFS.</li><li><strong>Inadequate query performance because of </strong><a href="https://www.unraveldata.com/common-failures-slowdowns-part-ii/#:~:text=the%20application%20increases.-,What%20is%20Data%20Skew%3F,-In%20an%20ideal"><strong>skewed data</strong></a><br>Data skewness can hamper the query performance. Spark tries to distribute an equal workload to all tasks when a query is executed. When Spark executes a query, specific tasks may get many small-size files, and the rest may get big-size files. For example, 200 tasks are processing 3 to 4 big-size files, and 2 are processing 1000 small-size files. Those two tasks take more time to finish processing than the other 200 tasks. This is how the query performance is impacted.</li></ul><h3>Solutions to &quot;the large number of small files&quot; issue</h3><p>We covered how a large number of small files can hamper the cluster and performance of the application. There are different ways to solve this problem.</p><ul><li>We can reduce the parallelism of an application to reduce the number of small files.</li><li>Also, we can optimize the dataframe before writing it to the object storage.</li></ul><p>Let&#39;s see how we can implement these solutions…</p><h4><strong>Reduce parallelism</strong></h4><p>When the Spark application is executed, it creates a <a href="https://www.projectpro.io/recipes/what-is-dag-apache-spark">DAG</a>. Then the <a href="https://books.japila.pl/apache-spark-internals/scheduler/DAGScheduler/">DAG Scheduler</a> creates stages, which are further divided into several tasks. A task is a unit of work that sends to the executor. The <a href="https://books.japila.pl/apache-spark-internals/scheduler/TaskScheduler/">Task Scheduler</a> schedules task execution on executors. Each stage has some task, one task per <a href="https://sparkbyexamples.com/spark/spark-shuffle-partitions/#:~:text=What%20is%20Spark%20Shuffle%3F">shuffle partition</a>. The Same task is done over different partitions of RDD. Each task processes each shuffle partition.</p><p>The number of shuffle partitions depends on two properties.</p><ul><li>spark.default.parallelism</li><li>spark.sql.shuffle.partitions</li></ul><p>The property spark.default.parallelism was introduced with <a href="https://spark.apache.org/docs/latest/rdd-programming-guide.html">RDD</a> hence this property is only applicable to RDD. This property defines the initial number of partitions created when creating an RDD. The default value for this configuration is set to the number of all cores on all nodes in a cluster. If you are running Spark locally, it is set to the number of cores on your system. You can set this property as below:</p><p>spark.default.parallelism = &lt;&lt;integer value&gt;&gt;</p><p>The property <a href="https://www.linkedin.com/pulse/always-more-resources-performance-spark-optimization-gaurav-patil">spark.sql.shuffle.partitions</a> controls the number of shuffle partitions created after <a href="https://sparkbyexamples.com/apache-spark-rdd/spark-rdd-transformations/#:~:text=of%20narrow%20transformation-,Wider%20Transformation,-Wider%20transformations%20are">wide transformation</a> for RDD and dataframe. Whenever wide transformations such as join(), agg(), etc., are executed by a Spark application, it generates <em>N</em> shuffle partitions where <em>N</em> is the value set by the spark.sql.shuffle.partitions property. Spark can potentially create <em>N</em> files in the output directory. If you are storing a dataframe in a partitioned table, the total number of files will equal <strong><em>N * &lt;potential table partitions&gt;</em></strong><em> </em>in the output directory<em>.</em> The default value for this property is 200. You should set the number of shuffle partitions according to your cluster&#39;s data size and available resources. The number of partitions should be multiple of the number of executors you have so that partitions can be equally distributed across tasks. You can set this property as below:</p><p>spark.sql.shuffle.partitions = &lt;&lt;integer value&gt;&gt;</p><p>Reducing the shuffle partitions value reduces small files because the same-size data gets distributed across the available shuffle partitions after decreasing shuffle partitions.</p><h4><strong>Reduce shuffle partitions before writing a dataframe</strong></h4><p>The simplest and most commonly used technique is to reduce shuffle partitions of the dataframe before saving it to the table. For repartitioning, Spark has two functions — <a href="https://sparkbyexamples.com/spark/spark-repartition-vs-coalesce">repartition()</a> and <a href="https://sparkbyexamples.com/spark/spark-repartition-vs-coalesce">coalesce()</a>. We can apply any one of the above functions to reduce the partitions. The coalesce() function works much faster when you reduce the number of partitions<strong> </strong>because it sticks input partitions together but coalesce() doesn&#39;t guarantee uniform data distribution. The repartition() function re-shuffles data across shuffle partitions, providing better distribution of data, but there is a lot of shuffling involved when using repartition() compared to coalesce(). Providing a column name along with the partition count to the repartition() function guarantees data distribution based on provided column values.</p><p>Let&#39;s see the difference between PySpark repartition() vs coalesce().</p><p>We will first create an RDD having 20 integer values.</p><pre>rdd = spark.sparkContext.parallelize(range(0,20))<br>rdd.collect() #show records in RDD</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/785/1*0UvMR0M5dD29qLqPb7X74g.png" /></figure><p>Let&#39;s use getNumPartitions() function to check the number of partitions of an RDD.</p><pre>print(&quot;Initial number of partitions: &quot; + str(rdd.getNumPartitions()))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IIKa9PZ8U7iz-yiScpqlsQ.png" /></figure><p>Now, we will print data in each partition. We can use the glom() function to check data present in each partition.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/980/1*NSnNYqgsr-vZFRzOTTg47g.png" /></figure><p>In the output above, you can see how RDD&#39;s data is distributed across different partitions.</p><p>Let&#39;s use the repartition() function on the first RDD we just created.</p><pre>rdd2 = rdd.repartition(4)<br>print(&quot;Number of partitions after repartition: &quot; + str(rdd2.getNumPartitions()))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*z7eIoog451hRqvoYgsIkfw.png" /></figure><p>Now, we will print data in each partition.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0VIGO5w07qcWVz6N7pdPwg.png" /></figure><p>Let&#39;s use the coalesce() function on the first RDD we created.</p><pre>rdd3 = rdd.coalesce(4)<br>print(&quot;Number of partitions after coalesce: &quot; + str(rdd3.getNumPartitions()))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*G5GDDykmgiJ3qa2GyTkcnQ.png" /></figure><p>Let&#39;s print data in each partition.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/965/1*EOGLDI3UVUUkWA3tNLgWZg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1003/1*PklMQvk5xISVMg6qB51gVw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/994/1*WWwFXUJaOVWjIRseVE27HA.png" /><figcaption>Visualization of the output</figcaption></figure><p>You can see the difference between records in partitions after using repartition() and coalesce() functions. Data is more shuffled when we use the repartition() function, whereas the coalesce() function does less shuffling. (It just moves data to the nearest partitions.) Also, we can see that the coalesce() function has a better distribution of data than repartition().</p><h4><strong>Repartition on a derived column</strong></h4><p>This is just an advancement to the above solution. In the above solution, the distribution of records across shuffle partitions is not guaranteed. Here we will create a derived column and pass it to the repartition function to redistribute data based on that column. This solution is applicable in the scenario where we don&#39;t have any column in which data is distributed evenly. In simple terms, our dataframe has skewed columns.</p><p>If you apply the repartition() function on the dataframe to reduce shuffle partitions to reduce the number of small files, you will observe that your Spark application has been slowed down. So, to resolve the small files issue without compromising performance, you need to create a derived column where values are highly distributed. You must pass this derived column along with the partition count to the function. After that, the repartition() function will equally distribute records across shuffle partitions. Then your small files issue will be solved without compromising the performance.</p><p>In the PySpark example<strong> </strong>given below, I am processing 2GB of data. I have a dataframe called df. I want to repartition it and then store it on a table. This can be achieved in three steps, as given below.</p><ol><li>Create a variable number_of_files and assign an integer value to it. Depending on the data, you need to tweak the value of this variable. I have set number_of_files = 10. Create a column having unique values, as shown below. I created the _unique_id column having unique values by using the monotonically_increasing_id() function.</li></ol><pre>from pyspark.sql import functions as F<br>number_of_files = 10<br>df = df.withColumn(&#39;_unique_id&#39;, F.monotonically_increasing_id())</pre><p>2. After this, apply the <a href="https://itnext.io/handling-data-skew-in-apache-spark-9f56343e58e8#:~:text=join%20cost%20itself.-,Key%20Salting">salting technique</a> to distribute records across partitions using a random value. In Spark, salting is a technique that adds random values to distribute Spark partition data evenly. For this, we need to create a derived column by taking the modulo value of the _unique_id column created above; not really a random number, but it works well. I have created a column called _salted_key in this example.</p><pre>df = df.withColumn(&#39;_salted_key&#39;, \<br>                           F.col(&#39;_unique_id&#39;) % number_of_files)</pre><p>3. Repartition dataframe based on _salted_key column.</p><pre>df = df.repartition(number_of_files, &#39;_salted_key&#39;) \                  <br>                           .drop(&#39;_unique_id&#39;, &#39;_salted_key&#39;)</pre><p>After the above three steps, we will have our dataframe repartitioned. We can save this dataframe to the target table or save it as a file.</p><h3>Conclusion</h3><p>In this article, I covered different techniques by which we can solve the &quot;large number of small files&quot; issue in Spark. There&#39;s no silver bullet<strong> </strong>in configuring Spark parameters for optimizing storage or application performance. It all depends on your data as well as the code. It is advised to perform some analysis on the source data beforehand to understand the data better so that you can try different techniques to optimize your Spark application&#39;s storage and performance.</p><p>Happy learning!</p><h3>References</h3><ul><li><a href="https://blog.cloudera.com/the-small-files-problem/">Small files problem</a></li><li><a href="https://itnext.io/handling-data-skew-in-apache-spark-9f56343e58e8">Handling data skewness in Apache Spark</a></li><li><a href="https://blog.rockthejvm.com/repartition-coalesce/">Repartition vs. Coalesce</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=21f819eb36d3" width="1" height="1" alt=""><hr><p><a href="https://medium.com/globant/how-to-solve-a-large-number-of-small-files-problem-in-spark-21f819eb36d3">How  to solve the “large number of small files” problem in Spark</a> was originally published in <a href="https://medium.com/globant">Globant</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Glance at Spark optimization techniques]]></title>
            <link>https://medium.com/globant/glance-on-spark-optimization-techniques-1d3df3074e41?source=rss-e04a068ef4ab------2</link>
            <guid isPermaLink="false">https://medium.com/p/1d3df3074e41</guid>
            <category><![CDATA[pyspark]]></category>
            <category><![CDATA[hadoop]]></category>
            <category><![CDATA[spark]]></category>
            <category><![CDATA[spark-optimization]]></category>
            <category><![CDATA[big-data]]></category>
            <dc:creator><![CDATA[Gaurav Patil]]></dc:creator>
            <pubDate>Mon, 31 Oct 2022 12:19:08 GMT</pubDate>
            <atom:updated>2022-12-07T13:20:15.970Z</atom:updated>
            <content:encoded><![CDATA[<h3>A Glance at Apache Spark optimization techniques</h3><p>A brief look at widely used performance optimization techniques for Spark</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*rtlRj__LHq0O9Vph" /><figcaption>Photo by <a href="https://unsplash.com/@marcojodoin?utm_source=medium&amp;utm_medium=referral">Marc-Olivier Jodoin</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Data is nothing but information about facts and figures. There are three primary sources of data: <a href="https://en.wikipedia.org/wiki/Social_data_analysis"><strong>Social data</strong></a>, <a href="https://en.wikipedia.org/wiki/Machine-generated_data"><strong>Machine-generated data</strong></a>, and <a href="https://en.wikipedia.org/wiki/Transaction_data"><strong>Transaction data</strong></a><strong>. </strong>Data analysis helps organizations to improve their internal processes, customer relationship management, increased user experience, potential improvements, etc. This vast data is nothing but <a href="https://en.wikipedia.org/wiki/Big_data">Big Data</a>. It needs high-speed computing techniques to process it faster because traditional systems cannot process it.</p><p><a href="https://spark.apache.org/"><strong>Spark</strong></a><strong> </strong>is an immensely powerful framework for processing big data. You might have seen that your Spark application runs slow, even though enough resources are allocated because that application might not be optimized enough. Spark has some in-built optimization techniques like <a href="https://www.databricks.com/glossary/catalyst-optimizer"><strong>Catalyst optimizer</strong></a> and <a href="https://www.databricks.com/glossary/tungsten"><strong>Tungsten optimizer</strong></a>, but there are more ways to optimize Spark applications.</p><p>This article is intended for data engineers and data scientists who work on Spark and want to gain some performance improvement in their Spark applications. In this article, I shall give you a brief understanding of some widely used optimization techniques supported by Spark and how you can use them. Let&#39;s start…</p><h3>Optimization techniques in Spark</h3><p>Spark engine has some built-in optimizations, but still, we need to focus more on other optimizations, which we might need to do in terms of cluster optimization, configuration values tweaks, table optimization, code level optimizations, etc., which collectively help to improve the performance of our application.</p><p>We can broadly categorize optimization techniques into three categories as given below:</p><ul><li><strong>Optimizing Spark configurations:</strong> This includes changing the Spark properties.</li><li><strong>Optimizing Spark program:</strong> This includes code-level optimizations.</li><li><strong>Optimizing storage:</strong> This includes file format optimization.</li></ul><p>Let&#39;s dive deep into each category…</p><h3>Optimizing Spark C<em>onfigurations</em></h3><p>Many Spark configuration properties are available in the Spark engine, which can be set while creating the SparkSesion object. These properties control the Spark application and can be tuned to improve the performance of the application.</p><p>How will you set the Spark configuration?</p><p>1. You need to import the required classes.</p><pre>from pyspark.sql import SparkSession</pre><p>2. Create an object and set the configuration value in the config() function.</p><pre>spark = (SparkSession.builder<br>           .master(&quot;yarn&quot;)<br>           .appName(&quot;test&quot;) <br>           .config(&quot;spark.sql.shuffle.partitions&quot;, 200)<br>           .getOrCreate())</pre><p>Now let&#39;s see which properties we can tune to improve the performance of the Spark application…</p><h4><strong>Tuning Spark Executor</strong></h4><p>Most of the time, we allocate static resources to the application. This strategy works well when we know our application&#39;s source data size. But what if some day you are getting massive data in source and the other day you are getting small data? Static allocation will not be a practical choice in this scenario. We can dynamically add/remove executors according to our Spark application&#39;s workload. Spark has a dynamic allocation technique to enable executor scaling. This approach is best when we have an inconsistent workload daily. You need to set the <em>spark.dynamicAllocation.enabled </em>property to &quot;true&quot; to enable the dynamic allocation of resources. After that Spark application will scale in and scale out executors according to its need.</p><p>This requires any one of the below two properties to be set:</p><ul><li><em>spark.shuffle.service.enabled</em></li><li><em>spark.dynamicAllocation.shuffle tracking.enabled</em></li></ul><p>The following properties are also relevant to <a href="https://spark.apache.org/docs/latest/configuration.html#dynamic-allocation">dynamic allocation configuration</a>:</p><ul><li><em>spark.dynamicAllocation.minExecutors</em></li><li><em>spark.dynamicAllocation.maxExecutors</em></li><li><em>spark.dynamicAllocation.initialExecutors</em></li><li><em>spark.dynamicAllocation.executorAllocationRatio</em></li></ul><h4><strong>Tuning Spark Memory</strong></h4><p>We often get an <a href="https://www.clairvoyant.ai/blog/apache-spark-out-of-memory-issue">out-of-memory error</a> either at the driver or executor sides. To avoid that, we need a correct memory configuration. The executor memory is divided into different layers, which can be tuned to improve performance. We need to <a href="https://stackoverflow.com/questions/39660377/spark-2-0-memory-fraction">set the <em>spark.memory.fraction</em></a> property to tune the values of these layers.</p><p>You can see three memory regions in the diagram below:</p><figure><img alt="Memory management in Spark" src="https://cdn-images-1.medium.com/max/531/1*ZLbiteNB9r_NnJm-1ufi6g.png" /><figcaption>Memory management in Spark</figcaption></figure><ul><li><strong><em>Reserved Memory: </em></strong>Memory reserved by the system and its size is hard coded. It is generally 300 MB.</li><li><strong><em>User Memory: </em></strong>User memory is calculated as<strong><em> </em></strong><em>(&quot;Java Heap Memory&quot; — &quot;Reserved Memory&quot;) * (1.0 — spark.memory.fraction).</em> This memory pool remains after the allocation of Spark Memory, and it is completely up to you to use it in the way you like. User Memory, and it&#39;s completely up to you what would be stored in this RAM and how Spark makes completely no accounting on what you do there and whether you respect this boundary or not. Not respecting this boundary in your code might cause an out-of-memory error.</li><li><strong><em>Spark Memory</em></strong>: Spark memory is calculated as <em>(&quot;Java Heap Memory&quot; — &quot;Reserved Memory&quot;) * spark.memory.fraction</em>. Memory pool, managed by Spark, is further divided into two memory regions — <em>Storage Memory </em>&amp;<em> Execution Memory.</em></li></ul><p><strong>Storage memory</strong> is used for storing all of the cached data, and broadcast variables are also stored here. Spark will store that data in this segment for any <em>persist()</em> option that includes &#39;MEMORY&#39;. It is calculated by using the formula given below:</p><p><em>Storage Memory = (Java Heap Memory — Reserved Memory) * spark.memory.fraction * spark.memory.storageFraction</em></p><p><strong>Execution memory </strong>is used by Spark for objects created during the execution of a task. It is calculated by using the formula given below:</p><p><em>Execution Memory = (Java Heap Memory — Reserved Memory) * spark.memory.fraction * (1.0 — spark.memory.storageFraction)</em></p><h4><strong>Tune Shuffle File Buffer</strong></h4><p>Disk access is slower when compared to in-memory data access as it involves a serialization process that takes up time and resources. We can reduce disk I/O costs by introducing a shuffle read/write file buffer in the memory.</p><p>The memory buffer size controls the disk seeks and system calls made in creating intermediate shuffle files. We need to set the <em>spark.shuffle.file.buffer</em> property to change the memory buffer size. The default value for this property is 32k. If the available memory resources are sufficient, we can increase the size of this parameter to reduce the number of times the disk file overflows during the shuffle write process, reducing the number of disk IO times and improving performance. The recommended size for this property is 1 MB. This allows Spark to do more buffering before writing the final map results to disk. We can set the property as given below:<br>spark.shuffle.file.buffer = 1 MB</p><h4><strong>Tune Compression block size</strong></h4><p>For large datasets, we can change the default compressed block size. These data blocks can be compressed through either storage or speed-based, like <a href="https://en.wikipedia.org/wiki/Lempel%E2%80%93Ziv%E2%80%93Oberhumer">LZO</a>, <a href="https://en.wikipedia.org/wiki/Snappy_(compression)">SNAPPY</a>, and <a href="https://en.wikipedia.org/wiki/Gzip">GZIP</a>. The below property needs to be set for the block size used in <a href="https://en.wikipedia.org/wiki/LZ4_(compression_algorithm)">LZ4 </a>compression. The default compression block size is 32 kb which is not optimal for large datasets. By increasing block size, you can see up to a 20% reduction in shuffle/spill file size. <br>spark.io.compression.lz4.blockSize = 512KB</p><figure><img alt="Compression block size vs size of shuffle files" src="https://cdn-images-1.medium.com/max/1024/1*3VwhXMRUXYjcbhxeJYr9Hg.png" /><figcaption>Compression block size vs. size of shuffle files</figcaption></figure><h4><strong>Tune Shuffle Partitions value</strong></h4><p>The shuffle-partition means the number of partitions generated after each transformation step that causes data shuffling, such as join(), agg(), reduce(), etc. By setting <em>spark.sql.shuffle.partitions</em> property, you can decide the level of parallelism in your Spark application.</p><p>You can <a href="https://www.linkedin.com/pulse/always-more-resources-performance-spark-optimization-gaurav-patil">set Spark property <em>spark.sql.shuffle.partitions</em></a> to control default shuffle partitions. The default value for this property is 200. You should set the number of shuffle partitions according to your cluster&#39;s data size and available resources. The number of partitions should be multiple of the executors you have so that partitions can be equally distributed across tasks. <br>spark.sql.shuffle.partitions = &lt;&lt;integer value&gt;&gt;</p><h4><strong>Use off-heap memory</strong></h4><p>Off-heap memory lies outside the JVM but is used by JVM for specific use cases (for example - interning of Strings). Spark can explicitly use Off-Heap memory to store serialized dataframes and RDDs. Spark uses off-heap memory for two purposes:</p><p>— A part of off-heap memory is used by Java internally for purposes like String interning and JVM overheads.<br> — Off-Heap memory can also be used by Spark explicitly for storing its data as part of Project Tungsten.</p><p>The total off-heap memory for a Spark executor is controlled by <em>spark.executor.memoryOverhead</em> configuration. The amount of off-heap memory used by Spark to store actual data frames is controlled by <em>spark.memory.offHeap.size</em>. It can be enabled by setting <em>spark.memory.offHeap.use</em> configuration to true. The difference between these two memories is explained <a href="https://stackoverflow.com/questions/61263618/difference-between-spark-executor-memoryoverhead-and-spark-memory-offheap-size">here</a>.</p><h3>Optimizing Spark Program</h3><p>The real power of a CPU can only be seen if we run a well-optimized program. We should not just think that increasing resources will help to improve performance unless we optimize our code. Spark provides many techniques we can include to optimize our application. This section will see some of the widely used program optimization techniques.</p><h4><strong>Broadcast Join</strong></h4><p>Let&#39;s consider a scenario where you are joining a big table with a small table. During this join operation, more <a href="https://sparkbyexamples.com/spark/spark-shuffle-partitions/#:~:text=Shuffling%20is%20a%20mechanism%20Spark,Disk%20I%2FO">shuffling</a> will happen. We can avoid shuffling by using a broadcast join. It will copy a small table to every node where the executor is running. However, after a certain threshold, broadcast join tends to have less advantage over shuffle-based joins.</p><pre># Example program to illustrate use of broadcast join<br>from pyspark.sql.functions import broadcast</pre><pre>emp_df = spark.sql(&quot;select id, name, dept_id from employee&quot;)<br>dept_df = spark.sql(&quot;select dept_id, dept_name from department&quot;)<br>df_joined = emp_df.join(broadcast(dept_df),emp_df.dept_id == dept_df.dept_id, ‘inner’)<br>df_joined.show(20)</pre><h4><strong>Cache data</strong></h4><p>Every time we call the <a href="https://sparkbyexamples.com/apache-spark-rdd/spark-rdd-actions/"><em>Action</em></a> in the Spark program, it triggers <a href="https://www.projectpro.io/recipes/what-is-dag-apache-spark">DAG</a> and executes it from the beginning. That&#39;s why it&#39;s recommended not to use unnecessary <em>Actions </em>in Spark programs. Double-check the code and remove or comment <em>Actions </em>you wrote for debugging/testing in your Spark program.</p><p>One key point to improve performance when the same <a href="https://spark.apache.org/docs/3.1.1/api/python/reference/api/pyspark.sql.DataFrame.html">dataframe</a> is being referred to in multiple places is to cache that dataframe. Spark has two functions to cache dataframe: cache() and persist(). RDD&#39;s cache() function default saves the dataframe to memory while the persist() function is used to store dataframe at the user-defined storage level. The persist() function supports different parameters.</p><figure><img alt="Storage Levels comparison chart" src="https://cdn-images-1.medium.com/max/1024/1*dwg2DnghjHx0QsZ1pJ-AoA.png" /><figcaption>Storage Levels comparison chart</figcaption></figure><h4><strong>Repartition data</strong></h4><p>A dataframe is partitioned means there are logical groups of records in it. A group of records is called a partition. Each task processes each partition, many tasks run in parallel inside one executor, and parallel execution happens in Spark. The level of parallelism can be increased if we distribute data correctly. For repartitioning, Spark has two methods — <a href="https://sparkbyexamples.com/spark/spark-repartition-vs-coalesce">repartition()</a> and <a href="https://sparkbyexamples.com/spark/spark-repartition-vs-coalesce">coalesce()</a>.</p><p>If we check the source code, we see that coalesce() is called inside the repartition()<em> </em>with the shuffle<em> </em>parameter set to true. The major difference between coalesce() without shuffle<em> </em>parameter<em> </em>and the repartition() is that coalesce() is mainly used to decrease partitions and repartition() is used to increase partitions. Less shuffling happens when we use coalesce() it compared torepartition(). The repartition() function can also be used to solve data skewness issues.</p><p>Below is the code snippet taken from the <a href="https://github.com/apache/spark/blob/15dc0eb6c6b5b5da4632eddafaec0ac82fe1043b/core/src/main/scala/org/apache/spark/rdd/RDD.scala#L473"><strong>RDD source code</strong></a>.</p><pre>/*<br>* Return a new RDD that has exactly numPartitions partitions.<br>* Can increase or decrease the level of parallelism in this RDD. <br>* Internally, this uses a shuffle to redistribute data.<br>* If you are decreasing the number of partitions in this RDD, <br>* consider using `coalesce`, which can avoid performing a shuffle.<br>*/<br>def repartition(numPartitions: Int)(implicit ord: Ordering[T] = null): RDD[T] = withScope {<br>    coalesce(numPartitions, shuffle = true)<br>}</pre><p>So, to improve the performance of your application, you need to either increase or decrease the partitions. That totally depends on your data. If you are processing large data and want to create more simultaneous tasks, then you can use the repartition() function to increase the partitions. If you want to re-shuffle the data based on column value, you can also use the repartition() function. If you want to reduce the number of output files, then you can use the coalesce() function. You can significantly improve the application&#39;s performance after applying these changes.</p><h4><strong>Filter data in earlier steps</strong></h4><p>The key point to improve the performance of joins and other processing is to filter data in earlier steps which you don&#39;t need in the result set. Let&#39;s say you want to process and store only employees in &#39;Mumbai&#39;. Then instead of reading the whole data and process and then applying filters on the result, you can just filter your dataframe earlier while reading the tables into the dataframe.</p><p>There is one more important feature of Spark. It optimizes the logical plan internally. It uses <a href="https://docs.datastax.com/en/dse/6.8/dse-dev/datastax_enterprise/spark/sparkPredicatePushdown.html">predicate pushdown</a> for supported file formats and pushes filters at the beginning, and then it does the processing on filtered data.</p><h4><strong>Use </strong><a href="https://towardsdatascience.com/skewed-data-in-spark-add-salt-to-compensate-16d44404088b"><strong>Salting Technique</strong></a><strong> to eliminate data skewness issues.</strong></h4><p>We often see on Spark UI that some tasks take longer, and some finish quickly. This happens when your data is skewed. That means data is not evenly distributed across the partitions. So, tasks working on partitions where the data size is more than the other partitions take more time to complete. This also causes an out-of-memory issue sometimes. Re-shuffling data across partitions can eliminate data skewness. We can achieve shuffling by using the repartition() function. But this function does not guarantee the even distribution of records. So, we must add some random values in a new column in the dataframe, which we often call the <em>salted key</em>, and then we can pass that <em>salted key</em> in the repartition function as an argument. After that repartition() function will re-shuffle data based on the <em>salted key</em> column values.</p><pre># Example program to illustrate use of Salting Technique</pre><pre>salt_df_new = df.withColumn(“salted_key”, <br>                                (rand * n).cast(IntegerType))<br>shuffled_df = salt_df_new.repartition(100, ‘salted_key’)</pre><h4><strong>Provide schema explicitly while reading data into dataframe</strong></h4><p>The predefined schema makes it easier for Spark to get columns and datatypes without reading the entire file; this improves the performance of your Spark code if you are dealing with a massive volume of data. We can use the inferSchema=True option if we want Spark to identify schemas implicitly, and we can use the schema option to provide a schema to dataframe.</p><pre># Example program showing how to provide the schema externally<br># while reading the file</pre><pre>from pyspark.sql.types import StructType, IntegerType, DateType</pre><pre>schema = StructType([<br>         StructField(“col_01”, IntegerType()),<br>         StructField(“col_02”, DateType()),<br>         StructField(“col_03”, IntegerType()) ])</pre><pre>df = spark.read.csv(filename, header=True, schema=schema)</pre><h4><strong>Optimize JSON files processing</strong></h4><p><a href="https://en.wikipedia.org/wiki/JSON">JSON</a> is a widely used format that allows for semi-structured data because it does not require a schema. This offers added flexibility to store and query data that doesn&#39;t always adhere to fixed schemas and data types. JSON files are compatible with many systems. It is always recommended to pass schema explicitly while reading JSON files to avoid incorrect identification of data types.</p><pre># Example program showing how to provide the schema externally <br># while reading JSON file</pre><pre>from pyspark.sql.types import StructType, StructField, StringType <br>schema = StructType([ <br>             StructField(“first_name”, StringType(), True),<br>             StructField(“last_name”, StringType(), True), <br>             StructField(“address”, StringType(), True)<br>         ])</pre><pre>emp_df = sqlContext.read.json(“employee_file.json”, schema)</pre><h4><strong>Use the ReduceByKey function over GroupByKey</strong></h4><p>Both reduceByKey() and groupByKey()<em> </em>are broad transformations, meaning both will shuffle across the partitions. The critical difference between reduceByKey() and groupByKey() is that reduceByKey()<em> </em>does a map side combine and groupByKey() does not. The reduceByKey() acts like a mini reducer. So, the shuffling of data can be reduced if we use reduceByKey().</p><figure><img alt="Reduce by key example" src="https://cdn-images-1.medium.com/max/861/1*Pcw6NlwwY9W12iDGUoK-XA.png" /><figcaption>Reduce by key example</figcaption></figure><figure><img alt="Group by key example" src="https://cdn-images-1.medium.com/max/860/1*3yhToDLcAtG2OJ6ArFSYzw.png" /><figcaption>Group by key example</figcaption></figure><h4><strong>Avoid the use of UDFs (User Defined Functions)</strong></h4><p>UDFs are heavy when it comes to performance. Spark does not optimize UDFs as it&#39;s like a black box for them. Built-in &quot;<a href="https://www.databricks.com/glossary/what-is-spark-sql">Spark SQL</a>&quot; functions are optimized and recommended to be used in programs. Spark provides a lot of commonly used Spark SQL functions for data transformation. So, before deciding to create a UDF, look at Spark SQL documentation to see if any function is available which can satisfy your requirement.</p><h4><strong>Check the uniqueness of keys while using the join columns</strong></h4><p>We should always carefully choose the join columns for joining two dataframes. If duplicate values exist in either of the columns, it takes a long time to join such dataframes because a cartesian product can happen. This also leads to an out-of-memory error. So, it is always better to analyze the table by executing some queries before we write the code.</p><h3>Optimizing Storage</h3><p>Optimizing storage is very important to improve the performance of the Spark application. This includes organizing data by creating partitions and buckets, using serialized data formats, choosing an appropriate file format according to the application&#39;s needs, etc. Let&#39;s see them one by one…</p><h4><strong>Bucketing and Partitioning</strong></h4><p>Bucketing and Partitioning are widely used techniques for optimizing hive tables. Partitioning is often represented as directories, and bucketing is represented as files. Partitioning splits records into files present under different directories named the same as partition column values, based on the partition column. Bucketing is helpful in further splitting records into different files based on a hashing function. This improves performance when we read data in Spark.</p><p>If the table is partitioned and bucketed, then the &quot;Spark SQL&quot; in which filter columns are the same as partitioned/bucketed columns can run faster because it only scans particular directories and files instead of scanning all directories. Thus, query performance is improved.</p><p>The<strong> cardinality</strong> of a column can be defined as the number of distinct values in that column. It is always recommended to use bucketing on high-cardinality columns and partitioning on low-cardinality columns.</p><pre># Example program showing how to use partitioning and bucketing <br># while creating a Hive table</pre><pre>CREATE EXTERNAL TABLE zipcodes(<br>    rec_no int,<br>    country string,<br>    city string,<br>    zip_code int<br>)<br>PARTITIONED BY(<br>    state string<br>)<br>CLUSTERED BY zip_code INTO 10 BUCKETS<br>ROW FORMAT DELIMITED<br>FIELDS TERMINATED BY ‘,’;</pre><h4><strong>Serialized data formats</strong></h4><p><strong>Serialization</strong><em> </em>is a mechanism for converting the state of an object into a byte stream. This mechanism is used to persist the object. <strong>Deserialization</strong><em> </em>is the reverse process where the byte stream is used to recreate the actual Java object in memory. Spark uses Java Serializer. We can explicitly hint Spark by using the <a href="https://blog.knoldus.com/kryo-serialization-in-spark/">Kryo serializer</a>, which improves performance.</p><p>Different file formats support serialization, like <a href="https://avro.apache.org/docs/1.2.0/">Avro</a>, <a href="https://www.databricks.com/glossary/what-is-parquet">Parquet</a>, and <a href="https://cwiki.apache.org/confluence/display/hive/languagemanual+orc#LanguageManualORC-ORCFiles">ORC</a>. These file formats also support compression libraries like <a href="https://docs.python.org/3/library/zlib.html">ZLIB</a>, <a href="https://en.wikipedia.org/wiki/Snappy_(compression)">SNAPPY</a>, etc.</p><h4>Choosing an appropriate file format</h4><p>Spark is compatible with many file formats such as text, Parquet, ORC, Avro, JSON, etc. Each file format has its advantages and disadvantages. Text format is widely used for unstructured data. The JSON format is used for semi-structured data. Structured data can be stored in ORC or Parquet, as both are columnar formats. ORC and Parquet are memory efficient and perform better in speeding up queries. Avro stores schema in JSON format, and it has broad compatibility with other systems. Depending on the requirement, we must choose the correct file format to balance performance and the need.</p><figure><img alt="File formats comparison" src="https://cdn-images-1.medium.com/max/1024/1*-Wx13r0kvYL1mUKQCvvWhQ.png" /><figcaption>File formats comparison</figcaption></figure><h3>Conclusion</h3><p>In this article, I gave you a brief understanding of all widely used optimization techniques. Though Spark is an immensely popular and prominent Big Data processing engine, optimization is always a challenging topic. Hence, it is essential to understand your data and all possible Spark configurations for your process.</p><p>There&#39;s no predefined way to optimize Spark applications. It all depends on your data and code. Make sure you perform different benchmarks and source data analyses before deciding on the optimization technique for your program. You can use the Spark UI to analyze your applications and determine where you can improve them. Thank you for reading my blog. Happy coding!</p><h3>References</h3><ul><li>Tuning Spark — <a href="https://spark.apache.org/docs/2.3.2/tuning.html">https://spark.apache.org/docs/2.3.2/tuning.html</a></li><li>Performance Tuning — <a href="https://spark.apache.org/docs/2.4.2/sql-performance-tuning.html">https://spark.apache.org/docs/2.4.2/sql-performance-tuning.html</a></li><li>Spark memory management — <a href="https://community.cloudera.com/t5/Community-Articles/Spark-Memory-Management/ta-p/317794">https://community.cloudera.com/t5/Community-Articles/Spark-Memory-Management/ta-p/317794</a></li><li>Big data file formats — <a href="https://www.clairvoyant.ai/blog/big-data-file-formats">https://www.clairvoyant.ai/blog/big-data-file-formats</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1d3df3074e41" width="1" height="1" alt=""><hr><p><a href="https://medium.com/globant/glance-on-spark-optimization-techniques-1d3df3074e41">Glance at Spark optimization techniques</a> was originally published in <a href="https://medium.com/globant">Globant</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>