<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Badwaik Ojas on Medium]]></title>
        <description><![CDATA[Stories by Badwaik Ojas on Medium]]></description>
        <link>https://medium.com/@badwaik.ojas?source=rss-6b750069b897------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*uHtFmxQvg7aReryd_xWDog.jpeg</url>
            <title>Stories by Badwaik Ojas on Medium</title>
            <link>https://medium.com/@badwaik.ojas?source=rss-6b750069b897------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 15:55:56 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@badwaik.ojas/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Lakehouse: A modern data Architecture]]></title>
            <link>https://medium.com/@badwaik.ojas/lakehouse-a-modern-data-architecture-9c7061169905?source=rss-6b750069b897------2</link>
            <guid isPermaLink="false">https://medium.com/p/9c7061169905</guid>
            <category><![CDATA[data-lakehouse]]></category>
            <category><![CDATA[data-lake]]></category>
            <category><![CDATA[data-warehouse]]></category>
            <category><![CDATA[data-architecture]]></category>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[Badwaik Ojas]]></dc:creator>
            <pubDate>Tue, 01 Aug 2023 17:12:38 GMT</pubDate>
            <atom:updated>2023-08-01T17:12:38.024Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="Pic Credits: Databricks" src="https://cdn-images-1.medium.com/max/1024/0*O6lWSdxR52yPg5Q7.png" /><figcaption>Different Data Architecture</figcaption></figure><p>Data always played an Important role in an Organization. It helped in knowing the Customer buying pattern(Retail), Patient response to the treatment(Healthcare), Fraud detection(Banking), Vaccine trials results(Pharmaceutical), Trending news(Social Media), Annual Sales of cars(Automobiles), Impact to Government Initiatives(Government), the list is endless.</p><p>With Internet boom as the data increased we started saving data in <strong>Databases</strong> and used this data for analytics purpose in decision making but as the data grew we realized that transactional databases have some limitations and are not optimized for the analytics workload, every department had their own database and it was difficult to have a birds eye view of all the data in organization(In order to make a decision, the user had to find WHICH version of the data was the right one to use among the many applications) etc. to rectify these limitations The <strong>Enterprise Data Warehouse</strong> were popularized. Th DW concept was introduced in 1990s these were specialized Databases which were optimized for Analytics workload which supported complex query, had the ability to stored historical data and had the overall view for the organizational data but it had some drawbacks like Data warehouses lock data into proprietary formats that increase the cost of migrating data or workloads to other systems. Given that data warehouses primarily provide SQL-only access, it is hard to run any other analytics engines, such as machine learning systems. Despite much research on the confluence of ML and data management, none of the leading machine learning systems, such as TensorFlow, PyTorch and XGBoost, work well on top of warehouses. Unlike BI, which extracts a small amount of data, ML systems process large datasets using complex non-SQL code.</p><p>For more details refer my Blog on <a href="https://medium.com/@badwaik.ojas/short-note-on-databases-data-warehouse-and-data-marts-5c87296bb273">Databases, Data Warehouse</a></p><p>Then came the Data boom(Big Data) which disrupted the Internet with data coming from different source like IoT, sensors, social media, clickbait&#39;s etc. But there was a problem with the data that was coming in, The data consist of Structured, Semi structured(JSON, CSV etc.) and Unstructured(Videos, Images etc.). Data Warehouse was used to stored the Structured data but had difficulty with the Semi structured and Unstructured data that was coming in.</p><p>To solve this problem <strong>Data Lake</strong> came for the help. The <strong>Data Lake</strong> is an amalgamation of ALL of the different kinds of data found in the corporation. It has become the place where enterprises offload all their data, given its low-cost storage systems with a file API that hold data in generic and open file formats, such as Apache Parquet and ORC. It was later realized that Data lake though seem useful but had issues like no support for transactions, no enforcement of data quality or governance and poor performance optimizations. As a result, most of the data lakes in the enterprise have become data swamps.</p><p>For more details refer my Blog on <a href="https://medium.com/@badwaik.ojas/short-note-on-data-lake-bf7a98fa21d1">Data Lake</a></p><p>There was a new Architecture that was then proposed where in it was a combination of Data Warehouse and Data Lake. More than 90% of enterprise data is stored in data lakes due to its flexibility from open direct access to files and low cost, as it uses cheap storage. To overcome the lack of performance and quality issues of the data lake, enterprises ETLed a small subset of data in the data lake to a downstream data warehouse for the most important decision support and BI applications. This dual system architecture requires continuous engineering to ETL data between the lake and warehouse. Each ETL step risks incurring failures or introducing bugs that reduce data quality, while keeping the data lake and warehouse consistent is difficult and costly. Apart from paying for continuous ETL, users pay double the storage cost for data copied to a warehouse.</p><p>To address the issue there emerged a new class of data architecture called <strong>Lakehouse</strong>.</p><h4>Data Lakehouse:</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/549/1*58o1ju7OJyFffQFz_0LFcg.png" /><figcaption>Lakehouse Architecture</figcaption></figure><p><strong>Lakehouse</strong> strives to combine the advantages of both the <strong>Data Lake</strong> and the <strong>DW</strong> paradigms while minimizing their challenges.</p><p><strong>Lakehouse</strong> enables a new open and standardized system design: implementing similar data structures and data management features to those in a data warehouse, directly on the kind of low cost storage used for data lakes. Lakehouse platform available in market <strong>Delta Lake</strong> from <strong>Databricks</strong>, <strong>Apache Iceberg</strong> which started at <strong>Netflix</strong>, <strong>Apache Hudi</strong> which started at <strong>Uber</strong>.</p><h4>Lakehouse Provides</h4><ol><li>Direct-access by using open formats, such as Apache Parquet.</li><li>First class support for data science and machine learning.</li><li>Offer best-in-class performance and reliability on low cost storage.</li></ol><h4>Lakehouse Feature</h4><ol><li><strong>Openness</strong>: The first key idea we propose for implementing a Lakehouse is to have the system store data in a low-cost object store (e.g., Amazon S3) using a standard file format such as Apache Parquet which Provides an open API that can efficiently access the data directly without the need for proprietary engines and vendor lock-in and Supports not only SQL access, but also a variety of other tools and engines, including machine learning and Python/R libraries.</li><li><strong>Transaction Support</strong>: Leverage ACID transactions to ensure consistency as multiple parties concurrently read or write data, typically using SQL.</li><li><strong>Low cost storage:</strong> Lakehouse architecture is built using low cost object storage such Amazon S3, Azure Blob Storage or Google Cloud Storage.</li><li><strong>Data quality enforcement and Governance:</strong> Support for data quality check and schema enforcement in order to keep the lake clean, maintain consistency of data and provide robust governance and auditing mechanisms.</li><li><strong>Performance optimizations:</strong> Enable various optimization techniques, such as caching, auxiliary data structure by leveraging file statistics, data layout strategies and data compaction to right-size the files.</li><li><strong>Support for machine learning and data science: </strong>ML systems’ support for direct reads from data lake formats already places them in a good position to efficiently access a Lakehouse. In addition, many ML systems have adopted DataFrames as the abstraction for manipulating data that enable performing query optimizations for data accesses in ML workloads.</li><li><strong>Storage is decoupled from compute:</strong> This means storage and compute use separate clusters. These systems are able to scale to many more concurrent users and larger data sizes.</li><li><strong>Streaming Support:</strong> Real-time reports are the norm in many enterprises. Lakehouse Support for streaming eliminates the need for separate systems dedicated to serving real-time data applications.</li></ol><h3>Main Components of Lakehouse</h3><p>The basic idea behind Lakehouse Architecture is provide Open format access, Low cost and Directly-accessible storage and providing traditional analytical DBMS management and performance features such as ACID transactions, data versioning, auditing, indexing, caching and query optimization.</p><h4>Metadata Layer over Data Lake</h4><p>The differentiating factor between a Data Lake and a Lakehouse is a Metadata layer on top of the Data Lake. This I believe is the first component in the Lakehouse Architecture.</p><p>This abstraction level is used to implement ACID transactions and other management features. Metadata layer stores the information about which objects are part of a table in the data lake itself as a transaction log in Parquet/ORC format, enabling it to scale to billions of objects per table. They also provide management features such as transactions, zero-copy coning and time travel to past versions of a table.</p><p>Metadata layers are a natural place to implement data quality enforcement features. For example, Delta Lake implements schema enforcement to ensure that the data uploaded to a table matches its schema, and constraints API that allows table owners to set constraints on the ingested data Delta’s client libraries will automatically reject records that violate these expectations or quarantine them in a special location. Hence improving the quality.</p><p>Metadata layers is were we implement governance features such as access control and audit logging. For example, a metadata layer can check whether a client is allowed to access a table before granting it credentials to read the raw data in the table from a cloud object store, and can reliably log all accesses.</p><h4>Data Processing</h4><p>The performance of the Data Processing framework in a Lakehouse depends on one of the many factors like hardware resources we have available, whether we can change the data object storage format instead of using existing standards such as Parquet and ORC.</p><p>Lakehouse has several techniques to implement performance optimizations in a Lakehouse independent of the chosen data format, which can therefore be applied either with existing or future formats. Techniques within the Databricks Delta Engine and show that they yield competitive performance with popular cloud data warehouses, though there is plenty of room for further performance optimizations. These format-independent optimizations are Caching, Auxiliary Data, Data Layout.</p><p><strong>Caching</strong>: Files that are cached from the cloud object store on faster storage devices such as SSDs and RAM on the processing nodes helps in improving the performance.</p><p><strong>Auxiliary Data: </strong>Auxiliary information or metadata, refers to additional information that is used to support or enhance the understanding, processing, or analysis of the main data. This supplementary data can provide context, descriptive details, or additional attributes that help make the primary data more meaningful and useful. Here it maintain column min-max statistics for each data file in the table within the same Parquet file used to store the transaction log, which enables data skipping optimizations when the base data is clustered by particular columns. It also implements a Bloom filter based index.</p><p><strong>Data layout</strong>: Data layout plays a large role in access performance. There are multiple layout decisions that can be optimized by the Lakehouse system. The most obvious is record ordering: which records are clustered together and hence easiest to read together. In Delta Lake for example it support ordering records using individual dimensions or space filling curves such as Z-order and Hilbert curves to provide locality across multiple dimensions.</p><p>These three optimizations work especially well together for the typical access patterns in analytical systems. In typical workloads, most queries tend to be concentrated against a “hot” subset of the data, which the Lakehouse can cache using the same optimized data structures as a closed-world data warehouse to provide competitive performance. For “cold” data the a cloud object store, the main determinant of performance is likely to be the amount of data read per query. In that case, the combination of data layout optimizations and auxiliary data structures such as zone maps (which let the engine rapidly figure out what ranges of the data files to read) can allow a Lakehouse system to minimize I/O the same way a closed-world proprietary data warehouse would, despite running against a standard open file format.</p><p>We can see that unified data platform architecture that implements data warehousing functionality over open data lake file formats can provide competitive performance with today’s data warehouse systems and help address many of the challenges facing data warehouse users.</p><p>Lakehouse has a great potential and is mostly likely will be the way our data architecture be designed around this concept. I believe that the industry is likely to converge towards Lakehouse designs given the vast amounts of data already in data lakes and the opportunity to greatly simplify enterprise data architectures.</p><p>Hope this blog helped you in better understanding of Data Lakehouse.</p><p>Happy Reading!!!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9c7061169905" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Short Note on Data Lake]]></title>
            <link>https://medium.com/@badwaik.ojas/short-note-on-data-lake-bf7a98fa21d1?source=rss-6b750069b897------2</link>
            <guid isPermaLink="false">https://medium.com/p/bf7a98fa21d1</guid>
            <category><![CDATA[big-data-analytics]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[big-data-and-analytics]]></category>
            <category><![CDATA[storage-solutions]]></category>
            <category><![CDATA[data-lake]]></category>
            <dc:creator><![CDATA[Badwaik Ojas]]></dc:creator>
            <pubDate>Thu, 27 Jul 2023 19:36:25 GMT</pubDate>
            <atom:updated>2023-07-27T19:36:25.571Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*t8soQB4MpKbSMJQI.jpg" /><figcaption>Data Lake</figcaption></figure><p>As you are aware there is a Data boom occurred a decade ago and non-relational data exploded fully, and there were no simple and easy-to-use systems in the market to handle that. Traditional database storage require data to be structured and organized before storage while Traditional file storage can be less scalable and may face limitations in managing large datasets efficiently. There was a need for storage mechanism that offer greater flexibility for storing and processing diverse data types in their raw form, making them ideal for exploratory analysis and data discovery also that can handle large volumes of data, making them suitable for big data scenarios.</p><p>Big Data emerged as a new trend as the volume, variety, and velocity (3Vs of Big Data) of data increased drastically. Hadoop was the option available to handle that situation at that time. It uses the Hadoop Distributed File System (HDFS) to store all types of data, including structured data, semi-structured data, and unstructured data.</p><p>This data needs a home, and the Data Lake is the preferred solution for creating that home. The term was invented and first described by James Dixon, CTO of Pentaho, who wrote in his blog: “If you think of a datamart as a store of bottled water — cleansed and packaged and structured for easy consumption — the Data Lake is a large body of water in a more natural state. The contents of the Data Lake stream in from a source to fill the lake, and various users of the lake can come to examine, dive in, or take samples.</p><p>Data Lake in simple words is any system that supports a file system like HDFS or allows you to store objects like BLOBs (Binary Large Objects). It is like a centralized repository that stores a large volume of raw, unprocessed, and diverse data in its native format.</p><p>Lets compare a real lake with a Data Lake to make it simpler to understand for you. As you know, a real lake receives water from its various tributaries it may also receive water from direct rain from the sky. Similarly, a Data Lake receives structured and unstructured data from various business applications, IoT devices, social media platforms, and so on. The data flowing into a Data Lake can be a real time stream of data or a batch of data coming in at a set frequency.</p><p>Under the hood Data Lake makes use of Distributed file systems they provide a scalable, fault-tolerant, and distributed storage infrastructure that spans across multiple nodes or servers. Eg: HDFS, Amazon S3 etc. Some Data Lakes utilize distributed NoSQL databases to store and manage unstructured and semi-structured data. NoSQL databases offer horizontal scalability and flexible data models. Examples include Apache Cassandra, Apache HBase, and MongoDB.</p><h4>Characteristics of a Data Lake</h4><p><strong>Scalability</strong>: Data Lakes are designed to handle vast amounts of data, making them highly scalable. As data volumes grow, organizations can easily expand the storage capacity of the data lake to meet their needs.</p><p><strong>Schema-on-Read</strong>: Unlike traditional databases that require a predefined schema before data insertion, data lakes adopt a “schema-on-read” approach. The schema is applied when data is read or queried, allowing users to apply different schema interpretations for different analysis purposes.</p><p><strong>Cost-Effectiveness</strong>: Data lakes can use cost-effective storage solutions like object storage or distributed file systems, making them more economical compared to traditional relational databases.</p><p><strong>Data Variety:</strong> Data lakes accept data from various sources and formats without any data transformation, enabling the storage of diverse data sets in their raw state.</p><p><strong>Data Exploration and Analytics</strong>: Data lakes provide a platform for data scientists, analysts, and other stakeholders to explore and analyze data in its raw form. This allows for more comprehensive and flexible data analysis, as users can perform various transformations and extract insights based on their specific requirements.</p><p><strong>Data Governance and Security</strong>: Data lakes implement robust data governance and access controls to ensure data security and compliance with regulatory standards. Access to data is controlled and monitored to protect sensitive information.</p><p><strong>Data Integration: </strong>Data lakes can serve as a central hub for data integration, enabling the consolidation of data from multiple sources, both internal and external to the organization.</p><p><strong>Real-time Data Processing: </strong>Modern data lakes can support real-time data streaming, allowing organizations to ingest and process data as it arrives, enabling timely insights and decision-making.</p><h4><strong>Challenges with Data Lake:</strong></h4><p>While Data Lakes always start out with good intentions, sometimes they take a wrong turn and end up as data swamps. A data swamp is a data pond that has grown to the size of a data lake but failed to attract a wide analyst community, usually due to a lack of self-service and governance facilities. Lets go through drawbacks Data Lake has.</p><p><strong>Data Quality and Governance Challenges</strong>: Storing data in its raw, unprocessed form can lead to data quality issues, such as missing, duplicate, or inconsistent data. Data governance becomes more complex as data lakes grow, making it challenging to maintain data accuracy, lineage, and compliance with regulations.</p><p><strong>Data Silos and Lack of Structure: </strong>Without proper organization and governance, data lakes can become data silos, where data is scattered and difficult to find. The lack of upfront schema definition may lead to data chaos and make it challenging to integrate and analyze data effectively.</p><p><strong>Complexity in Data Processing and Analytics: </strong>Analyzing data in its raw form may require additional data preparation steps, transformation, and cleansing before it can be used for analytics. This complexity can result in longer data processing times and increased analytical efforts.</p><p><strong>Scalability and Performance Concerns:</strong> Data lakes can become challenging to manage and scale as data volumes grow exponentially. The increased data volume can impact data retrieval and processing performance, necessitating the need for sophisticated hardware and distributed computing.</p><p><strong>Increased Storage Costs:</strong> Although data lakes offer cost-effective storage options, storing massive amounts of raw data over time can still result in significant storage costs, especially when using cloud-based storage solutions.</p><h4>Addressing Data quality issues in Data Lake</h4><p>Handling data quality issues and ensuring data consistency in a data lake environment requires a systematic approach and a combination of data governance practices, data validation, and data cleansing techniques. Lets discuss few.</p><p><strong>Data Governance and Data Quality Policies:</strong> Establish data governance policies that define data quality standards, guidelines, and best practices. Clearly outline data quality requirements and expectations for data ingested into the data lake.</p><p><strong>Metadata Management and Data Lineage: </strong>Implement robust metadata management to capture information about data sources, data transformations, and data lineage. Understanding the data lineage helps trace data back to its origin and identify potential data quality issues.</p><p><strong>Data Profiling:</strong> Conduct data profiling on the incoming data to assess its quality, completeness, and consistency. Data profiling tools can help identify patterns, anomalies, and data discrepancies.</p><p><strong>Data Validation Rules:</strong> Define data validation rules to check the integrity and accuracy of data during the data ingestion process. These rules verify data against predefined criteria to identify any data quality issues.</p><p><strong>Data Cleansing and Transformation:</strong> Implement data cleansing and transformation processes to address data quality issues. Use techniques like data standardization, deduplication, and data normalization to improve data consistency and accuracy.</p><p><strong>Data Quality Monitoring: </strong>Continuously monitor data quality metrics and data quality trends. Establish data quality monitoring dashboards and alerts to proactively identify and address data quality issues.</p><p><strong>Data Retention and Data Archiving:</strong> Implement data retention policies and data archiving strategies to manage data lifecycle effectively. Regularly clean up obsolete or redundant data to maintain data consistency and reduce data clutter.</p><h4><strong>Data Lake Use Case:</strong></h4><p>let’s consider a large retail chain that operates both stores and an online e-commerce platform. The organization faced several challenges related to data analysis and decision-making due to the vast amount of data generated from various sources, including sales transactions, customer interactions, inventory management, marketing campaigns, and web analytics.</p><p>Before implementing a data lake, the retail chain encountered the following issues:</p><p><strong>Data Silos</strong>: Data was stored in separate systems and databases, leading to data silos and making it difficult to gain a holistic view of the business.</p><p><strong>Data Latency</strong>: Data processing was slow, causing delays in generating insights, which impacted the organization’s ability to respond quickly to market trends and customer demands.</p><p><strong>Data Variety</strong>: The retail chain dealt with structured data from transactional systems, unstructured data from customer feedback and social media, and semi-structured data from web logs and clickstream data. Traditional databases struggled to handle this variety of data efficiently.</p><p><strong>Limited Scalability</strong>: As data volumes grew, traditional databases struggled to scale, leading to performance bottlenecks and increased costs.</p><p>To address these challenges, the retail chain decided to implement a data lake. Here’s how the implementation led to significant improvements in data analysis and decision-making:</p><p><strong>Data Centralization and Unification</strong>: All data sources, including sales data, customer interactions, inventory data, and web analytics, were ingested into the data lake. Data was now centralized in a unified repository, eliminating data silos.</p><p><strong>Real-time Data Streaming and Processing</strong>: The data lake incorporated real-time data streaming from sales transactions and website interactions. Real-time data processing and analytics allowed the organization to monitor sales and customer behavior in real-time, enabling timely adjustments to marketing strategies and inventory management.</p><p><strong>Scalable Storage and Processing</strong>: The data lake, based on a distributed file system and big data processing frameworks like Apache Spark, provided the scalability needed to handle the growing data volume and perform complex data analytics efficiently.</p><p><strong>Data Analytics and Insights</strong>: Data scientists and analysts gained access to the data lake, enabling them to perform advanced analytics, customer segmentation, and market basket analysis. The data lake’s flexibility allowed them to explore diverse data types and derive valuable insights.</p><p><strong>360-Degree Customer View</strong>: With data from multiple channels and touchpoints stored in the data lake, the organization achieved a 360-degree view of customer behavior, preferences, and sentiments, leading to better customer understanding and personalized marketing strategies.</p><p><strong>Predictive Analytics and Forecasting</strong>: The data lake’s integration with machine learning libraries allowed the organization to implement predictive analytics models for demand forecasting, inventory optimization, and fraud detection.</p><p><strong>Data-Driven Decision-Making</strong>: By providing timely, accurate, and relevant insights, the data lake empowered business stakeholders to make data-driven decisions. Real-time insights into sales performance, customer behavior, and market trends enabled the retail chain to quickly adjust pricing, promotions, and inventory levels.</p><p><strong>Cost Efficiency</strong>: The organization achieved cost efficiency by leveraging cloud-based object storage for the data lake. It reduced the need for costly hardware infrastructure while scaling resources according to demand.</p><p>Implementing the data lake, the retail chain experienced a significant improvement in data analysis and decision-making. They gained a deeper understanding of customer preferences, optimized inventory management, and enhanced marketing strategies, leading to increased customer satisfaction and revenue growth. The data lake became a powerful platform for data-driven innovation and competitive advantage in the retail industry.</p><h4>The Do’s</h4><p>Implementing a data lake requires careful planning and consideration to ensure its success and effectiveness.</p><ol><li>Have clear business understanding and use cases defined and Understand the specific problem you are trying to solve using Data Lake.</li><li>Develop a comprehensive data governance framework for the Data Lake. Define data ownership, data quality standards, access controls, data retention policies, and compliance requirements.</li><li>Identify the data sources and understand the data integration and ingestion requirements.</li><li>Decide between on-premises storage, cloud-based storage, or a hybrid approach. Select the distributed file system or cloud storage service that suits your scalability and performance requirements.</li><li>Pay careful attention to data lake security. Implement robust access controls, encryption mechanisms, and authentication protocols to protect sensitive data from unauthorized access and ensure compliance with data privacy regulations.</li><li>Develop a data catalog that documents information about the data sources, data formats, data definitions, and data lineage.</li><li>Assess the required skill sets and expertise needed to manage the data lake.</li><li>Evaluate the costs involved in setting up and maintaining the data lake. Consider both the upfront costs and ongoing operational costs. Assess the potential return on investment (ROI) and the value the data lake will bring to the organization.</li><li>Prepare for organizational change management, as the adoption of a data lake may require cultural shifts in how data is handled and used. Promote a data-driven culture within the organization.</li></ol><p>Data lakes have become increasingly popular as organizations strive to leverage their data assets fully. By providing a scalable and flexible data repository, data lakes facilitate advanced analytics, machine learning, and data-driven decision-making processes.</p><p>When data lakes first came onto the scene, a lot of companies rushed out to buy Hadoop clusters and fill them with raw data, without a clear understanding of how it would be utilized. This led to the creation of massive data swamps with millions of files containing petabytes of data and no way to make sense of that data.</p><p>It’s important to note that while data lakes offer great flexibility, they also require thoughtful planning and management to avoid turning into “data swamps” where data becomes disorganized and difficult to interpret. Implementing proper metadata management, data governance policies, and data cataloging helps maintain the data lake’s usefulness and accessibility.</p><p>Hope this helped you to have a better understanding of Data Lake.</p><p>Thanks for reading!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bf7a98fa21d1" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Short Note on Databases, Data Warehouse and Data Marts]]></title>
            <link>https://medium.com/@badwaik.ojas/short-note-on-databases-data-warehouse-and-data-marts-5c87296bb273?source=rss-6b750069b897------2</link>
            <guid isPermaLink="false">https://medium.com/p/5c87296bb273</guid>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[data-warehouse]]></category>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[data-mart]]></category>
            <category><![CDATA[database]]></category>
            <dc:creator><![CDATA[Badwaik Ojas]]></dc:creator>
            <pubDate>Thu, 29 Jun 2023 09:17:41 GMT</pubDate>
            <atom:updated>2023-06-29T09:17:41.385Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/860/0*jTp-9Oz6oE4GcC_4" /></figure><p>From the earliest days of computers, storing and manipulating data have been a major application focus. Today, more than at any previous time, the success of an organization depends on its ability to acquire accurate and timely data about its operations, to manage this data effectively, and to use it to analyze and guide its activities. Phrases such as the information superhighway have become ubiquitous, and information processing is a rapidly growing multibillion dollar industry. The amount of information available to us is literally exploding, and the value of data as an organizational asset is widely recognized.</p><p>Yet without the ability to manage this vast amount of data, and to quickly able to extract the information that is relevant to a given question, as the amount of information increases, it tends to become a distraction and a liability, rather than an asset.</p><p>When the Computer Era began we started to save data on files in Computers and used tools like Spreadsheet applications, such as Microsoft Excel, to widely store and inspect data. It was easy to sort the data in different ways, and see the features and patterns in the data just by looking at it. This worked well as long as the data size was small.</p><p>Just imagine what it would be like if a large company, with tens of thousands of customers, with all the customer list in a simple spreadsheet. In a big company, it’s likely that several people would need to update the list, here the problem with concurrency came in. Suppose we also wanted to store details of each order a customer placed. We could start putting order information next to each customer, but as the number of orders per customer grew, the spreadsheet would get more and more complex. Suppose there are multiple sheets one having customer, product, order data and we want to find the connection between the sheets the solution we get would be quite complex. As the data grew in volume became difficult to manage these data in file. The tools we had were outdated and did not supported to handle the request that came in.</p><p>There was a need for increasingly powerful and flexible data management systems. To get the most out of their large and complex datasets, users must have tools that simplify the tasks of managing the data and extracting useful information in a timely fashion. Otherwise, data can become a liability, with the cost of acquiring it and managing it far exceeding the value that is derived from it.</p><p><strong>Databases</strong> came to the rescue. A database management system, or DBMS, is software designed to help in maintaining and utilizing large collections of data. As DBMSs entered the Internet Age it gained importance as more and more data is brought on-line, and made ever more accessible through computer networking.</p><h4>Advantages of <strong>Databases</strong>:</h4><ol><li><strong>Databases</strong> provides an abstract view of the data and hides complex details like data representation and storage details from the applications and users.</li><li><strong>Databases</strong> provides an efficient way to store data using compressing algorithms, normalization etc. also provide efficient way to retrieve data using techniques like partitioning, indexing.</li><li><strong>Databases</strong> provides various Integrity and Security features so data accessed from DBMS would be only be accessible to user who have the access to that. It was also provides various Encryption feature to protect your data.</li><li><strong>Databases</strong> provides centralized data administration for all the data in the database. These helps professionals with experience manage different groups and users, use techniques to minimize redundancy, fine tune the storage for efficient query execution.</li><li><strong>Databases</strong> provides concurrent access to the data such that multiple users can access the data at the same time.</li><li><strong>Databases</strong> helps in reducing the development time.</li></ol><p><strong>Databases</strong> seems promising technology that helped with most of the issues we faced earlier. It is mostly used for OLTP(On-line Transaction Processing) type of loads. The OLTP paradigm is focused on transactions. In the e-commerce website database, a simple transaction could involve entering a new order, user querying know the status of a given order etc. a typical OLTP query like the above would require accessing only a few records of the database. Since OLTP systems must support heavy transaction loads, their design should prevent update anomalies, and thus, OLTP databases are highly normalized.</p><p>There was dot-com boom that happened in 1990s and there was a massive flow and collection of data that was happening, organizations realized that they needed to perform sophisticated data analysis to support their decision-making processes.</p><p>Organizations are facing increasingly complex challenges in terms of management and problem solving in order to achieve their operational goals. This situation compels people in those organizations to use analysis tools that can better support their decisions.</p><p>The traditional <strong>Databases</strong> did not supported requirements for data analysis since they were designed and optimized to support daily business transactions, providing concurrent access to the system to different users, APIs etc., ensuring consistency etc. Apart from this traditional <strong>Databases</strong> systems do not include historical data, performance is not good for complex queries. Furthermore when there is requirement to perform analysis for data across different business units in an organization it becomes a difficult task.</p><p>There was need for new paradigm specifically oriented to analyze the data to support decision making. On-line Analytical Processing (OLAP), is focused on analytical queries. OLAP-oriented databases should support a heavy query load. A Typical OLAP queries over the e-commerce website database would ask, for example, for the total sales amount by product and by customer, or for the most ordered products by customer. These kinds of queries involve aggregation, and thus, processing them will require, most of the time, accessing all the records in a database table, Normalization is not good for these queries, since it partitions the database into many tables. Reconstructing the data would require a high number of joins.</p><p>Therefore, the need for a different database model to support OLAP was clear and led to the notion of <strong>Data Warehouse</strong>.</p><p><strong>Data Warehouse</strong> are large repositories that consolidate data from operational databases and other data sources and transforms data into new structures that fit better for the task of performing business analysis. They are dedicated analysis databases, <strong>Data Warehouses</strong> are designed and optimized to efficiently support OLAP queries. They are developed and deployed as an integral part of decision-support systems to provide an infrastructure that enables users to obtain efficient and accurate responses to complex queries.</p><p><strong>Data Warehouse</strong> focus on the analytical needs of different areas of an organization. These areas vary depending on the kind of activities performed by the organization. For example, in a retail company, the analysis may focus on product sales or inventory management. In <strong>Operational</strong> <strong>Databases</strong>, on the contrary, the focus is on specific functions that applications must perform, for example, registering sales of products or inventory status.</p><h4>Operational Databases vs Data Warehouses</h4><ol><li>Operational Databases is designed and optimized for small and simple transactions which are carried out frequently and repeatedly. Data access for OLTP requires. For example, in the e-commerce website database, a user may frequently insert new orders, modify old ones, and delete orders. Thus, the number of records accessed by it is usually small. On the other hand Data Warehouses must support complex queries, thus requiring access to all the records in one or more tables, resulting in long, complex SQL queries. These systems are less frequently accessed than Operational Databases. Example, a system handling purchase orders is frequently accessed, while a performing analysis of orders may not be that frequent</li><li>Operational Databases usually have a short query response time, provided the appropriate indexing structures are defined, while complex Data Warehouse queries can take longer time to complete.</li><li>Operational Databases have normally a high number of concurrent accesses and require locking or other concurrency management mechanisms to ensure safe transaction processing. On the other hand, OLAP systems are read only, and thus queries can be submitted and computed concurrently. Also, the number of concurrent users in an OLAP system is usually low.</li><li>Operational Databases models lead to a highly normalized schema, adequate for databases that support frequent transactions, to guarantee consistency and reduce redundancy. Data Warehouses have denormalized database schema, with a high level of redundancy, which favors query processing</li></ol><p>A key difference between operational <strong>Databases</strong> and <strong>Data Warehouses</strong> is the fact that, in the latter, data are extracted from several source systems. Thus, data must be transformed to fit the data warehouse model, and loaded into the data warehouse. This process is called extraction, transformation, and loading (ETL), and it has been proven crucial for the success of a data warehousing project.</p><p>A <strong>Data Warehouse</strong> is aimed at analyzing the data of an entire organization. It is often the case that particular departments of an organization only require a portion of the organizational data warehouse for their needs. For example, a sales department may only need sales data, while a human resources department may need demographic data and data about the employees. These departmental data warehouses are called <strong>Data Marts</strong>.</p><p>This is all about Databases, Data Warehouse and Data Marts to get you started and know the basic difference between them.</p><p>Hope you liked it.</p><p>Thanks for Reading!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5c87296bb273" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Memory Management in Apache Spark]]></title>
            <link>https://medium.com/@badwaik.ojas/memory-management-in-apache-spark-e3bbc76eecc5?source=rss-6b750069b897------2</link>
            <guid isPermaLink="false">https://medium.com/p/e3bbc76eecc5</guid>
            <category><![CDATA[optimization]]></category>
            <category><![CDATA[apache-spark]]></category>
            <category><![CDATA[garbage-collection]]></category>
            <category><![CDATA[memory-management]]></category>
            <category><![CDATA[storage]]></category>
            <dc:creator><![CDATA[Badwaik Ojas]]></dc:creator>
            <pubDate>Mon, 08 May 2023 15:35:35 GMT</pubDate>
            <atom:updated>2023-05-08T15:35:35.186Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*PQbeMJFSjLKBLQpw.png" /></figure><p>Apache Spark’s performance advantage over MapReduce is greatest in use-cases involving repeated computations. Much of this performance increase is due to Spark’s use of in-memory persistence. Rather than writing to disk between each pass through the data, Spark has the option of keeping the data on the executors loaded into memory. That way, the data on each partition is available in-memory each time it needs to be accessed.</p><p>If you do a quick Internet search on common Spark application performance issues, you discover they broadly fall into two categories: OOM (out of memory) and taking a long time to complete. When this happens, your Spark application stops working or crashes and the only thing you can do is figure out the underlying cause, fix it and restart your application. To overcome these performance/OOM issues, it requires understanding the underlying mechanisms i.e. <strong>Memory Management</strong>.</p><p>In this blog lets see what is Memory Management in Apache Spark, what are different options available, what affects the memory management and things that we need to take care while working.</p><p><strong>Memory Management </strong>in Apache Spark is way by which we Configure or Manage the memory of the <strong>Spark Application </strong>and<strong> Spark Cluster </strong>in such a way as to avoid above mentioned issues.</p><p>Spark Application consist of Driver and Executor. Lets discuss the Memory configurations of these.</p><h4>Driver Memory</h4><p>In a Spark Application Driver memory is set using the configuration “<strong>spark.driver.memory</strong>” whose default is 1GB. This is usually specified when starting a Spark application via the spark-submit.</p><p>The driver is mainly responsible for orchestrating Spark application’s workload, so it doesn’t need as much memory as executors. There are two scenarios when it needs to allocate memory, and any incorrect usage of Spark in these scenarios likely causes an OOM issue.</p><p>The first scenario is when either the RDD.collect or DataFrame.collect action is invoked and the driver memory is less than RDD or Dataframe.</p><p>The second scenario is when Spark is trying to broadcast the content of a dataset. You encounter an OOM issue if the data of the Dataframe that is to be broadcasted is too big.</p><h4>Executor Memory</h4><p>The memory management on the Spark executor side is more involved and complicated than on the driver side.</p><p>Executor memory is set using “<strong>spark.executor.memory</strong>” the default value of this is 1GB, But this value does not include overhead so the spark executor will require more space than this value specified.</p><p>Overhead is space is the Amount of additional memory to be allocated per executor process, in MiB unless otherwise specified. This is memory that accounts for things like VM overheads, interned strings, other native overheads, etc. This tends to grow with the executor size. Value of this Overhead is defined by configuration “<strong>spark.executor.memoryOverhead</strong>” and “<strong>spark.executor.memoryOverheadFactor</strong>”</p><pre>spark.executor.memoryOverheadFactor = 0.10 (default)<br>spark.executor.memoryOverhead = <br>  MaxOf(executor_memory*spark.executor.memoryOverheadFactor, 384M)</pre><p>“<strong>spark.executor.memory</strong>” also called execution memory is required to compute transformation and caching. The size used for execution and caching together is governed by configuration “<strong>spark.memory.fraction</strong>” which is 0.60 by default. the rest (1–0.60 = 0.40) is reserved for Spark’s internal metadata and user data structures.</p><p>Out-of-memory errors during a transformation or when cached partitions are spilling to disk, is usually caused by the limitation in this combined storage and execution space.</p><p>Within the “<strong>spark.memory.fraction</strong>” we have both execution and storage space. Storage in this case refers to Spark’s in-memory storage of partitions, whether serialized or not. Rather than providing a fixed region for storage, Spark allows applications that do not cache anything in-memory to use the full memory fraction for execution. Within the execution space, Spark defines a region of the execution for storing cached data. The size of this is determined by a percentage of set by the “<strong>spark.memory.storageFraction” </strong>(default value = 0.50). In Storage space Spark will not reclaim for execution if there is cached data present. Math goes as below.</p><pre>spark.executor.memory -&gt; user provided<br>spark.memory.fraction -&gt; default to 0.60<br>spark.memory.storageFraction -&gt; default to 0.50<br><br>Execution Memory = spark.executor.memory <br>                          * spark.memory.fraction <br>                          * (1 - spark.memory.storageFraction)<br><br>Storage Memory = spark.executor.memory <br>                          * spark.memory.fraction <br>                          * spark.memory.storageFraction</pre><p>The maximum memory size of container to running executor is determined by the sum of “<strong>spark.executor.memoryOverhead</strong>”, “<strong>spark.executor.memory</strong>” and “<strong>spark.executor.pyspark.memory</strong>” (when using pyspark).</p><p>These calculation help us to plan you job based on the dataset we intend to process in the Spark Application.</p><p>These are the basics w.r.t the memory being assigned to the Spark’s driver and executor for executing Spark Application.</p><h4>Storage</h4><p>RDD and Dataframes are stored using the options like in-memory as deserialized data, in-memory as serialized data, and on disk. Each has its features.</p><ol><li><strong>In-Memory as deserialized data: </strong>This form of in-memory storage is the fastest, since it reduces serialization time; however, it may not be the most memory efficient, since it requires the data to be stored as objects.</li><li><strong>In-Memory as serialized data: </strong>Using the standard Java serialization library, Spark objects are converted into streams of bytes as they are moved around the network. This approach may be slower, since serialized data is more CPU-intensive to read than deserialized data. however, it is often more memory efficient, since it allows the user to choose a more efficient representation. While Java serialization is more efficient than full objects, Kryo serialization can be even more space efficient.</li><li><strong>On Disk: </strong>RDDs, whose partitions are too large to be stored in RAM on each of the executors, can be written to disk. This strategy is obviously slower for repeated computations, but can be more fault-tolerant for long sequences of transformations, and may be the only feasible option for enormous computations.</li></ol><p>The persist() function in the RDD class lets the user control how the RDD is stored. Persist function allows parameter to be passed in it. Using this parameter we can configure different ways in which the RDD can be persisted in the Spark Application.</p><p>Check the memory consumption by the RDD, way to do this cache the RDD and look at the Spark UI’s storage tab for the details for memory consumed. To estimate the memory consumption by the object use <strong>SizeEstimator’s estimate</strong> method to get an idea.</p><pre>size = SizeEstimator.estimate(rdd)</pre><p>Another way to find the size of the cached Dataframe or RDD is using below code.</p><pre>size =<br>    spark.sessionState<br>      .executePlan(df.queryExecution.logical)<br>      .optimizedPlan<br>      .stats<br>      .sizeInBytes</pre><p>Looking at the size of the Dataframe and RDD we can get a fair idea of the storage consumption in the executor memory and help us plan forward or rectify the application code as to optimize.</p><p>Different application has different needs, some application would be there where you read the data only once and process it and in some application where in we would be performing multiple iteration on the same data in both the case our approach would be different but it is good to have the knowledge as discussed it helps us to make intelligent decisions.</p><h4><strong>Garbage Collection</strong></h4><p>It is the process of freeing up the memory allocated for an object once that object is no longer needed. Since Spark runs in the JVM, which has automatic memory management and large data structures, garbage collection can quickly become an expensive part of our Spark job. Garbage collection errors are a common cause of failure.</p><p>Even if garbage collection overhead doesn’t prohibit a job from running, garbage collection creates additional serialization time, which can significantly slow it down. We can minimize the GC cost by reducing the number of objects and the size of those objects. We can reduce the size and number of our objects by reusing existing objects and by using data structures (such as primitive types) that take up less space in memory.</p><p>The cost of garbage collection is proportional to the number of Java objects, so using data structures with fewer objects (e.g. an array of Ints instead of a LinkedList) greatly lowers this cost. An even better method is to persist objects in serialized form.</p><p>Please refer this <a href="https://spark.apache.org/docs/3.4.0/tuning.html#garbage-collection-tuning">spark documentation</a> for details on GC.</p><p>With this we come to an end for Memory Management in Apache Spark. Hope you get what you were looking for.</p><p>Thanks for Reading!!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e3bbc76eecc5" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Job Scheduling in Apache Spark]]></title>
            <link>https://medium.com/@badwaik.ojas/job-scheduling-in-apache-spark-e7b2ce2fbd?source=rss-6b750069b897------2</link>
            <guid isPermaLink="false">https://medium.com/p/e7b2ce2fbd</guid>
            <category><![CDATA[apache-spark]]></category>
            <category><![CDATA[fifo]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[resource-utilization]]></category>
            <category><![CDATA[job-scheduling]]></category>
            <dc:creator><![CDATA[Badwaik Ojas]]></dc:creator>
            <pubDate>Sun, 07 May 2023 08:45:09 GMT</pubDate>
            <atom:updated>2023-05-07T08:45:09.023Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*VPHhPNdpK-s8PF7J.png" /></figure><p>To give you a context about what we would be talking about in this blog. A <strong>Spark application</strong> consists of a driver process, which is where the high-level Spark logic is written, and a series of executor processes that can be scattered across the nodes of a cluster. The Spark program itself runs in the driver node and sends some instructions to the executors. One <strong>Spark cluster</strong> can run several Spark applications concurrently. The applications are scheduled by the cluster manager and correspond to one SparkContext. Spark applications can, in turn, run multiple concurrent jobs.</p><p>In short Apache Spark job scheduling can be done at the Cluster level or at Spark Application level.</p><p><strong>Cluster Level</strong>: Here a job refers to an multiple Spark Application and this refers to scheduling jobs on the same cluster based on the resources available in the cluster. Here each job requests for resources to the cluster manager for processing the job. Once the request is approved, these resources are locked and won’t be available for subsequent job until they are released.</p><p><strong>Application Level</strong>: In Spark Action triggers a Job, In this case this is what we are talking about. There might be case where there are multiple jobs request that comes in, job scheduling refers to scheduling these jobs accordingly.</p><h3>Job Scheduling at Cluster Level</h3><p>Job Scheduling here boils down to Resource availability in Cluster. When you run a Spark Application on a cluster it is given a specified number of resources and these resources won’t be shared with other applications and if there are multiple users/applications find need to share the cluster then there are 2 defined ways in Spark of allocating and managing resources across users/application.</p><h4><strong>Static Allocation</strong></h4><p>In this a Spark Application is allocated a fixed number of resources in the cluster and these resources are booked for the time till the Application is running. These resources are released once the Application ends and is available for next Application to run. This approach is available in Spark’s Standalone, mesos(Note: won’t be talking about mesos in this blog) and YARN cluster manager.</p><p>There are different options available using which we can control the number of resource to be allocation for the spark application like for <strong>Spark’s Standalone</strong> mode by default the application running will try to consume all the resource available but this can be controlled using “<strong>spark.cores.max</strong>” to the value you find would be enough for the application(PS: try testing with different value before coming to some value) also use “<strong>spark.executor.memory</strong>” to set the executor memory. When this value is not set at that time it would use “<strong>spark.deploy.defaultCores</strong>” which is set to infinite meaning max available in the cluster. In YARN when submitting Spark Application we use parameters like <strong>num-executor</strong>, <strong>executor-memory</strong>, <strong>executor-cores</strong> to control resource per executor. When can set the same parameters using the conf with parameters like “<strong>spark.executor.memory</strong>”, <strong>“spark.executor.cores” </strong>and<strong> “spark.executor.instances”</strong>.</p><h4><strong>Dynamic Allocation</strong></h4><p>Dynamic resource allocation expands the functionality of static allocation. In dynamic allocation, executors are added and removed from a Spark application as needed, based on a set of heuristics for estimated resource requirement. This feature is useful when multiple application share the same resources in Spark Cluster. This feature is disabled by default. lookout below configuration to use when using Dynamic Allocation “<strong>spark.dynamicAllocation.enabled</strong>”, “<strong>spark.dynamicAllocation.minExecutors</strong>”, “<strong>spark.dynamicAllocation.maxExecutors</strong>”, “<strong>spark.dynamicAllocation.initialExecutors</strong>”.</p><p>Other important configs like “<strong>spark.shuffle.service.enabled</strong>”, “<strong>spark.dynamicAllocation.shuffleTracking.enabled</strong>”. The purpose of this configs(shuffle tracking or the external shuffle service) is to allow executors to be removed without deleting shuffle files written by them.</p><p><strong>Add Request</strong> is made for additional executors when it has pending tasks waiting to be scheduled. This condition necessarily implies that the existing set of executors is insufficient to simultaneously saturate all tasks that have been submitted but not yet finished. Spark requests executors in rounds. configurations to lookout for is “<strong>spark.dynamicAllocation.schedulerBacklogTimeout</strong>” , “<strong>spark.dynamicAllocation.sustainedSchedulerBacklogTimeout</strong>”.</p><p><strong>Remove Request</strong> is made for removing executors is much simpler. A Spark application removes an executor when it has been idle for more than “<strong>spark.dynamicAllocation.executorIdleTimeout</strong>” seconds.</p><p>There might be cases where in the you are running the Spark application with Dynamic allocation. Resources are added as there were pending tasks and after the tasks were completed these extra resources that were added were no longer needed and were removed. Now, what about the state that would have been stored in the executor, the shuffle files written to these executors, As if these states are lost they would need to be recomputed and this would be an issue.</p><p>Apache Spark has the solution for this where in you can use the configuration “<strong>spark.shuffle.service.enabled</strong>” when this is enabled the spark executor will fetch the shuffle files from the service instead of from each other. Shuffle state will be available beyond executors lifetime.</p><p>Now regrading the data which are cached and persisted. The default behavior is executors containing cached data are never removed. There is conf like “<strong>spark.dynamicAllocation.cachedExecutorIdleTimeout</strong>”, “<strong>spark.shuffle.service.fetch.rdd.enabled</strong>” that can be used for this issue. Spark will use the External service to fetch the persisted data. Future implementation it is thought that these cached data would be stored in the off-heap storage so as to make it available.</p><h3>Job Scheduling at Application Level</h3><p>Spark is capable of running multiple jobs in a application provided they are requested from different threads. In this case the resources are allocated by one of the above discussed process. A job in here refers to any Action in Spark. Sparks scheduler is also thread safe and supports application that server multiple request.</p><p>By default, Spark schedules jobs in first in, first out (FIFO) to queue jobs within a system. This means that the first job submitted will run in its entirety, getting priority on all the available resources. However, if a job doesn’t need the whole cluster, the next job may start. FIFO scheduling can be useful to ensure that space-intensive jobs are able to use the resources that they need. However, if you launch a job a few seconds behind a many-hour process, the FIFO strategy can be frustrating. Spark offers a fair scheduler, modeled after the Hadoop fair scheduler, to allow high-traffic clusters to share resources more evenly. The fair scheduler allocates the tasks from different jobs to the executors in a “round-robin fashion” (i.e., parsing out a few tasks to the executors from each job). With the fair scheduler, a short, small job can be launched before an earlier long-running job is completed. However, The process of distributing these tasks is done by the TaskScheduler and varies depending on whether the fair scheduler or FIFO scheduler is used.</p><p>Now talking about fair scheduler it ensures that jobs get a more even share of cluster resources. The Spark application then launches jobs in the order that their corresponding actions were called on the SparkContext.</p><p>The fair scheduler also supports putting jobs into pools and allocating different priority (weight) to those pools. Jobs within a pool are allocated the same number of resources, and the pools are allocated resources according to their weight. Using pools can be a good way to ensure that high-priority jobs or very expensive jobs are completed.</p><p>This feature is disabled by default. To enable the fair scheduler use “<strong>spark.scheduler.mode</strong>” conf and set it to “<strong>FAIR</strong>”.</p><p>The fair scheduler also supports grouping jobs into pools, and setting different scheduling options (e.g. weight) for each pool. This can be useful to create a “high-priority” pool for more important jobs.</p><p>Lookout for <a href="https://spark.apache.org/docs/latest/job-scheduling.html#fair-scheduler-pools">Spark documentation</a> for configuring these pools.</p><p>Job scheduling comes in handy when working in sharing environment where we run multiple spark application on the same cluster or run multiple request comes in for a single application it helps us use the resources intelligently without affecting the end execution.</p><p>With this we come to an end to this topic. Hope this helps.</p><p>Thanks for reading!!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e7b2ce2fbd" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Tuning/Optimizing Apache Spark Job]]></title>
            <link>https://medium.com/@badwaik.ojas/tuning-optimizing-apache-spark-job-d784176e67d8?source=rss-6b750069b897------2</link>
            <guid isPermaLink="false">https://medium.com/p/d784176e67d8</guid>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[optimization]]></category>
            <category><![CDATA[apache-spark]]></category>
            <category><![CDATA[tuning]]></category>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[Badwaik Ojas]]></dc:creator>
            <pubDate>Tue, 02 May 2023 19:21:59 GMT</pubDate>
            <atom:updated>2023-05-02T19:21:59.601Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*UG5kLPubfW-SmEi-.png" /></figure><p>Sometimes just knowing the tools are not important, knowing how to use it plays most important part. Though we have tools to process Big Data it can be quite cumbersome at times to process the data. Sometimes even a small data can be troublesome to process by a cluster and sometimes even a big data can be processed by a small cluster if optimized correctly.</p><p>Apache Spark a big data processing engine even though very fast when compared to its preceder MapReduce sometimes needs Optimization/Tuning the job in order to execute efficiently and faster.</p><p>In this blog lets go through some of the Optimization/Tuning we can implement.</p><p>There are different part in the entire job execution cycle where we can perform optimization these include. Each has its own tricks and methods. Finding the right combination with each part is important for efficiently processing the job. I have divided this execution cycle where the Optimization/Tuning can be applied in 2 parts.</p><ol><li>When the data is laid out on disk and consumed in job.</li><li>When processing the data and performing actions</li></ol><h3>When the data is laid out on disk and consumed</h3><p>This refer to selecting layout of the dataset, the file format used, compression to be used, data locality and cluster configuration.</p><h4>Cluster Configuration</h4><p>To consume the big data file and perform the different operations on the data it should be processed on the cluster of machines in parallel for efficiency and speed. The configurations of the machines, the type of machines used in the cluster, number of executors to request, memory for executor etc. will the point to discuss in this case as in the end its the cluster that is going to process the big data.</p><p>Please refer one of my blog where I have explained the details of <a href="https://medium.com/@badwaik.ojas/spark-cluster-creation-things-to-consider-cf0cb9f33f1c">cluster creation</a> for your big data job.</p><h4>Partitioning</h4><p>You must have heard about partitions in Databases, in Big Data world we have a concept called HIVE Partitions. This is like a partition of data based on certain values in the dataset.</p><p>Consider for example I want to read a dataset into spark and then perform different transformation for analysis purpose.</p><p>Observe the DAG for the same dataset and same code when there is no partition and when there is partition in the dataset. This considerably very small dataset and the time difference you can see to evaluate the code is high imagine the difference it could make with working with Terabyte of code.</p><pre>  spark<br>    .read<br>    .schema(&quot;id Int, &quot; +<br>      &quot;name String, &quot; +<br>      &quot;age Int, &quot; +<br>      &quot;deptid Int, &quot; +<br>      &quot;salary Int&quot;)<br>    .csv(&quot;/path/to/emp_dataset&quot;).filter(&quot;deptid =1&quot;)<br>    .collect()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/690/1*gfD7hH6yjhKsmoz3wRogrA.png" /></figure><p>Hive Partitioning plays an important role when working with Big Data it pushes the filter down to the disk level and fetches only the data that is required and hence reducing the data ingestion in Spark engine and less the data faster the results and execution.</p><p>Though we have some advantages with Partitioning but there are few draw backs associated with these. If the data is not big enough the difference would not be that considerable and differentiating factor. Over partitioning gives rise to the tiny file problems. When having partition is then becomes a important to use the partition column in the query as if we don’t use the column in the query it would have to scan all the directory and files to fetch the result giving considerable performance lag to the job execution.</p><p>Experiment with both the stuff by having partition and without partition for your job and check the usage and performance under both implement the same which best suits your case.</p><h4>File Format</h4><p>The total size of the dataset the most important characteristics of data is its volume. Since volume becomes critical in systems for processing and storing big data, it becomes necessary to search for such a format that would have the ability to store data with minimum volumn.</p><p>There are different file format available in Big Data world, each has its features. Popular file format includes CSV, JSON, AVRO, Parquet, ORC.</p><p>Raw data is generally stored in text format such as CSV, Text file, JSON, XML. These are also known as human-readable formats, i.e., they can be read and edited by humans. However, the issue with raw data formats is, it takes a tremendous amount of storage space. Therefore, it becomes crucial to compress and store the files.</p><p>ORC, Avro, Parquet formats are designed for systems that use Spark/MapReduce kinds of framework, and it is a structure that is a systematic combination of multiple components, including data storage format, data compression, and optimization techniques for data reading. Furthermore, the Avro and Parquet data formats offer schema evolution.</p><p>When developing system architectures for Big Data, the question of choosing the optimal data storage format may arise.</p><p><strong>Avro</strong> is a row-oriented data storage format. It contains a schema in the JSON format, which allows faster reading and interpretation operations. The file structure consists of a header and data blocks. Avro format supports primitive types, such as Boolean, Int, long, float, etc. and complex types, such as array or map. Avro creates a binary structured format that is both compressible and split able. As a result, it can be efficiently used to input Spark/Hadoop MapReduce jobs.</p><p>Avro can easily handle changes in schema such as addition or changes in existing fields and thus, new programs can scan old data, and old programs can scan new data. Avro is extremely useful in systems where we see data is written continuously.</p><p><strong>Comma-separated values (CSV)</strong> is a textual format describing data in form of a table. A CSV file does not support different data types and structures — all data are presented as strings.</p><p><strong>JavaScript object notation(JSON)</strong> is a simple text format. JSON has gained popularity in storing big data in document databases. JSON supports data types and structures, such as string, number, Boolean, arrays, null, internal objects.</p><p><strong>Optimized row columnar(ORC)</strong> is a column-oriented storage format. Data in ORC are strongly typed. ORC has a shared internal structure — division into strips independent from each other. ORC files contain metadata storing in compressed forms, and include statistical and descriptive information, indexes, stripe, and stream information. ORC supports a complete set of types, including complex types (structures, lists, maps, and unions). ORC also complies with ACID requirements by adding delta files.</p><p>It can store data in an optimized way than the other file formats. Original data can be reduced by about 75% due to this and hence there is an increase in data processing speed and shows better performance than other file formats discussed.</p><p>It has various advantages over other file formats. It involves storing columns separately, storing statistics (Min, Max, Sum, Count), having a lightweight index, skips blocks of the row that are not part of the query.</p><p><strong>Apache Parquet</strong> is a column-oriented binary format. It allows defining compression schemes at the column level and adding new encodings as they appear. Parquet supports simple (Boolean, int32, float, etc.) and complex (byte_array, map) data types. The Parquet file contains metadata written after meaningful data to provide a one-pass write.</p><p>The idea behind columnar format is straightforward instead of storing the records in row by row, store the record using the column, which is beneficial for analytical processing. Multiple types of data can be converted and stored into Parquet format. Parquet supports well-organized compression and encoding schemes. Parquet supports efficient columnar data representation available to any project in the Hadoop ecosystem, regardless of the choice of data processing framework, data model, or programming language. The main advantage of Parquet is that a query can read and perform on all values for a column while reading only a small fraction of the data from a data file or table.</p><p>Avro supported row-oriented data access provides better performance on scan queries, e.g. when all columns are of interest, but Parquet/ORC format should provide a better performance on column-oriented queries, e.g. when only a subset of those are selected.</p><p>There are few Important Spark Configurations and Features that are important and work only with specific formats(ORC, JSON, Parquet) like predicate pushdown(Where we push the column names and filter conditions to the file level that will only retrieve the relevant data). Spark Configurations like “<strong>spark.sql.files.maxPartitionBytes</strong>”, “<strong>spark.sql.files.openCostInBytes</strong>”, “<strong>spark.sql.files.minPartitionNum</strong>” which can be useful when reading the file. Please check the Official Spark documentation for the details.</p><p>Choosing a file format is very important in your spark job as it is the matter of performance and SLA but it also depends on your use case. When your use-case involve calculating total salary of each employee in the department then Columnar format seems the better choice. When your use case involve fetching the complete details of employee Row based format seems the best choice.</p><h4>Others issues like Tiny Files, Scanning</h4><p><strong>Tiny Files issues? what exactly is this?</strong> with the number of records and size you would see have significant difference. As for consideration like for 40million records in few 100 files would have small size as compared to 40million records in lakhs small size files.</p><p>Tiny files come with overhead of a lot of Open and Close operations. Total Scan time required to read the tiny files is very high when compared to the small number of big files. File system read time for a large number of small files is more when compared to a small number of big files.</p><p>Way to reduce the Tiny files problem is to merge the files into fewer files once a particular point is reached.</p><p>Avoid over partitioning, suppose you are partitioning over a date is fine avoid partition on hour as this might create more files and tiny files with less data or no data.</p><p><strong>Can Scanning be an issue? Oh yes!</strong><br>Over partition can lead to scanning issues. Scanning issue is delay in reading the dataset. When we read the dataset since there is no action performed for this we expect this to be fast but suppose there is a delay there might exist a scanning problem. What exactly it does is when you execute the read on the dataset it browses through the directory for the files and then registers it even though no action is performed.</p><p>What happens when we specify the schema?<br>Even though we specify the schema, here we eliminate the schema read time but since we have many files and directory it would still take time to scan it.</p><p><strong>Way to reduce the scanning problem is to register it as a table.</strong> By registering the database table it stores the dataset details in the metastore. Metastore not only keeps track of various metrics such as number of rows min, max. But the table itself knows the location of all the files so it doesn’t have to go back and check. It does solve the performance problem. As a data engineer this would be the first time performance problem that you would get. But it would be beneficial for Data Scientist and Data Analyst.</p><h3>When processing the data and performing actions</h3><p>When data is loaded and we start processing it depending on what you propose to do with it there are many parts that might need to be optimized. Suppose you have data which is skewed, There is spill to disk when processing, challenges with shuffling, how the data is stored, serialization that can be used, the number of partitions(PS: this is different from the one discussed before), you might want to do multiple iterations with the data, what about garbage collection etc.</p><p>Based on what challenges you are facing there are different techniques to mitigate the problem. lets start to discuss those.</p><p>For any Big data job when we try to run those, if those job are not running as expected the first step is check if the issue is among the 5S.</p><p>5S stands for Spill, Skew, Shuffle, Storage, Serialization.</p><h4>Spill</h4><p>This refers to writing the temp files during processing to the disk when the RAM is insufficient to hold the data. This occurs when the given partition is too large to fit into the RAM memory. This is generally done to avoid the OOM issues.</p><p>The issue occurs when we have a wide transformation those partitions are too large, Use the explode operation, Use join or crossjoin etc.</p><p>The way to know if there is any Spill in the job is to look at the Spark Web UI -&gt; Stage Details page &gt; Under Summary Metrics.</p><p>Ways to reduce Spill is to allocate more memory per worker, if we have transformation that induce skewness then that needs to be addressed, partition size could be reduced using spark configuration parameter like “ <strong>spark.sql.shuffle.partitions</strong>”, “<strong>spark.sql.files.maxPartitionBytes</strong>”, Repartition can be used to redistribute the data.</p><h4>Skew</h4><p>This refers to the imbalance of data across the partitions. Data is typically read equally across all the machines in cluster but there might be transformation in the job like group by which will add significantly more records in one partition when compared with other.</p><p>We also assume when I say the data is distributed equally it is not exact to be precise there is always skew but it is typically so small that that it can be ignored and won’t affect the performance and execution of the job.</p><p>Issue that comes with Skew is the executors takes much time/resource to process the the data as there are more records as compared with others as result since it requires time to process the time required to complete the job is increased due this particular skew partition.</p><p>Skewness if any can be seen in the Spark UI -&gt; Stage Details -&gt; Event Timeline task distribution and time consumed.</p><p>There are different techniques used to reduce Skewness in the dataset like salting, hints, repartitioning etc.</p><p>With Spark 3 we also have features like DPP and AQE to mitigate the skew issue.</p><h4>Shuffle</h4><p>This refers to the act of moving data between executor. Transformations like group by, distinct, join order by etc. induce a wide transformation which move the data across the executor to achieve the goal.</p><p>The shuffle is also an integral part of the Big data job where working with 2 or more dataset are common and in such cases shuffle cannot be avoided, But there are ways to reduce the pain that a shuffle operation can induce.</p><p>Configuration like “<strong>spark.sql.shuffle.partitions</strong>” is one of that. default value of this is 200. This refers to the number of partitions at the end of the shuffling phase. Set this value to an appropriate value too high value mean to many partitions after the shuffling phase with less or no data too low value mean big partitions and less parallelization.</p><p>The idea is to analyze the size of your data and divide it into the a value that you think is enough to shuffle without congesting the network IO.</p><p>Other ways to mitigate the downside is to use fewer and larger executors hence reducing the network IO.</p><p>Reduce the data using filter, selecting only the columns that are needed can also help as less the data less would be an issue shuffling.</p><p>Use the Broadcast Hash join if the dataset you think is not big enough and can be broadcasted across the executor.</p><p>Bucketing can also be useful to mitigate the issues that can come up with shuffling. Bucketing here pre eliminates the exchange and sort by pre shuffling the data. Data is pre aggregated into N buckets.</p><h4>Storage</h4><p>I have discussed most of the storage related issue in above section. But there remains one thing that needs mention in this part.</p><p>Schema, Schema Evolution and Merging.</p><p>Inferring the schema requires a full read of the file to determine the data type, even if you only want a subset of data. Reading the parquet file requires one time reading of the schema. Supporting schema evolution is expensive as it has to read the schema of all the available files and then merge it.</p><p>The problem can be solved explicitly providing the schema.</p><h4>Serialisation</h4><p>Serialization is the process of converting a data object into a series of bytes that saves the state of the object in an easily transmittable form.</p><p>Serialization is used when transferring data over the network. Spilling data to disk. Caching memory to serialized. Broadcast variables.</p><p>Serialization plays an important role in the performance of any distributed application. Formats that are slow to serialize objects into, or consume a large number of bytes, will greatly slow down the computation. Often, this will be the first thing you should tune to optimize a Spark application.</p><p>There is Java Serialization and Kyro Serialization. Java serialization is flexible but often quite slow, and leads to large serialized formats for many classes. Kyro serialization is the way to go for as it is significantly faster and more compact than Java serialization (often as much as 10x), but does not support all serializable types and requires you to register the classes you’ll use in the program in advance for best performance.</p><p>use conf.set(“spark.serializer”, “org.apache.spark.serializer.KryoSerializer”) to register Kyro Serialization.</p><h4>Repartition and Coalesce</h4><p>As we move with our spark Job execution data might increase due to operations like join, crossjoin, explode etc. the data size might increase and we might redistribute the data in order to avoid spill, skew. Data might decrease due to operations like filtering there we might need to reduce the partition to be shuffled in order to avoid heavy network IO.</p><p>In all these scenario we might use Repartition and Coalesce to manage partition to be shuffled.</p><p>Checkout my blog on <a href="https://medium.com/@badwaik.ojas/repartition-vs-coalesce-in-apache-spark-e51b42a832c">Repartition and Coalesce</a> to know more.</p><h4>Cache, Persist and Checkpoint</h4><p>When the job works on Iterative algorithm that same dataset it does not make sense to go to disk level to fetch the data for each action. It would be better we cache the dataset so that it would be available for faster access. Cache and Persist can play and bigger role in this.</p><p>As we move forward with our iterative algorithm and perform multiple transformation. the Spark Execution Graph grows bigger and bigger and there might be case it is so big that we would get OOM errors, there might be case we want to save the data at that particular point in our execution cycle before moving forward so as to avoid re execution of the data. Checkpointing can most helpful in such cases.</p><p>Using these feature is limited to iterative algorithm where in it uses the same dataset again and again. Sometimes these features can becomes the roadblock where in it cripples the catalyst optimizer and does not allows to query the data at disk but only limits itself to the cached version.</p><p>Checkout my blog on <a href="https://medium.com/@badwaik.ojas/persist-cache-and-checkpoint-in-apache-spark-ae71783ce3dd">Cache, Persist and Checkpoint</a> to know more.</p><h4>Garbage Collection</h4><p>This refers to the process of freeing up the memory allocated for an object once that object is no longer needed. Since Spark runs in the JVM, which has automatic memory management and large data structures, garbage collection can quickly become an expensive part of our Spark job. Garbage collection or “GC” errors are a common cause of failure. Even if garbage collection overhead doesn’t prohibit a job from running, garbage collection creates additional serialization time, which can significantly slow it down.</p><p>GC can be a problem when you have large number of RDD stored. When Java needs to evict old objects to make room for new ones, it will need to trace through all your Java objects and find the unused ones. The main point to remember here is that the cost of garbage collection is proportional to the number of Java objects, so using data structures with fewer objects (e.g. an array of Ints instead of a LinkedList) greatly lowers this cost. An even better method is to persist objects in serialized form, the first thing to try if GC is a problem is to use serialized caching.</p><p>We can minimize the GC cost by reducing the number of objects and the size of those objects. We can reduce the size and number of our objects by reusing existing objects and by using data structures (such as primitive types) that take up less space in memory.</p><p>You can use multiple garbage collectors to evict the old objects and place the new ones into the memory. However, the latest Garbage First Garbage Collector (G1GC) overcomes the latency and throughput limitations with the old garbage collectors.</p><p><strong>Set the value for executor using</strong>:<br>“spark.executor.extraJavaOptions”: “-XX:+UseG1GC -XX:+UnlockDiagnosticVMOptions -XX:+G1SummarizeConcMark -XX:InitiatingHeapOccupancyPercent=35 -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:OnOutOfMemoryError=’kill -9 %p’”</p><p><strong>Set the value for Driver using</strong>:<br>“spark.driver.extraJavaOptions”: “-XX:+UseG1GC -XX:+UnlockDiagnosticVMOptions -XX:+G1SummarizeConcMark -XX:InitiatingHeapOccupancyPercent=35 -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:OnOutOfMemoryError=’kill -9 %p’”</p><p><strong>Details of parameter</strong>:<br><strong>XX:+UseG1GC</strong>: This specifies G1GC garbage collector to use default is-XX:+UseParallelGC<br><strong>-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps</strong>: This specify the frequency and execution time of the garbage collection.<br><strong>XX:InitiatingHeapOccupancyPercent</strong>: set this to a number you think is enough and occupying this percent of memory will initiate garbage collection.</p><p>Finally we come to an end to this topic but I am sure there is no end to this. The more you dig the more you find. The is truly and interesting topic and curiosity just gets in you to check for the area you can improve your job.</p><p>Thanks for reading my Blog!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d784176e67d8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Spark Cluster Creation: Things to consider?]]></title>
            <link>https://medium.com/@badwaik.ojas/spark-cluster-creation-things-to-consider-cf0cb9f33f1c?source=rss-6b750069b897------2</link>
            <guid isPermaLink="false">https://medium.com/p/cf0cb9f33f1c</guid>
            <category><![CDATA[apache-spark]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[cluster]]></category>
            <category><![CDATA[spark-cluster]]></category>
            <dc:creator><![CDATA[Badwaik Ojas]]></dc:creator>
            <pubDate>Tue, 25 Apr 2023 21:06:12 GMT</pubDate>
            <atom:updated>2023-05-03T20:13:43.300Z</atom:updated>
            <content:encoded><![CDATA[<h3>Spark Cluster: Things to consider?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*t6Oax0Q4UiGCCoPC.png" /></figure><p>To run our Apache Spark job we would need a machine that would be able to handle the Job we are creating in order to consume/transform the data. There arises questions like what kind of machines we would be using for our cluster, how much cores and memory to assign, where the cluster would reside etc. Apart from this there are many things to consider like what will the cluster be used for, who will be using the cluster, were to host(on-prem or cloud), what would the cost and where is the data residing and data size we create the cluster. In this blog lets discuss these things.</p><p>PS: When I am explaining below. most of the configuration I am telling w.r.t YARN as resource manager but most of these configurations are somewhat applicable to other also.</p><h3>Guide to num-executor, executor-cores, executor-memory</h3><p>Resource allocation is an important aspect when running a Spark Job. The resource allocation if done correctly helps run the Spark Job smoothly and if not done properly will consume the entire cluster resource and make other application deprive of resources, slow down your execution, sometimes fail to start spark job and might give OOM issues.</p><p>Consider the below Spark-Submit command</p><pre>spark-submit –num-executors ? –executor-cores ? –executor-memory ?..</pre><p><strong>num-executors:</strong> This argument only works on YARN and Kubernetes only. The value indicates the number of executors to launch. By default, the value is 2.</p><p><strong>executor-cores:</strong> This argument only works on Spark standalone, YARN and Kubernetes only. The value indicates the number of cores used by each executor. The default is 1 in YARN. HDFS client has trouble with tons of concurrent threads. It was observed that HDFS achieves full write throughput with ~5 tasks per executor . So it’s good to keep the number of cores per executor below that number.</p><p><strong>executor-memory: </strong>This argument represents the memory per executor (e.g. 1000M, 2G, 3T). The default value is 1G. spark.yarn.executor.memoryOverhead = Max(384MB, 7% of spark.executor-memory)</p><p>Full memory requested to yarn per executor = spark.executor.memory + spark.yarn.executor.memoryOverhead.</p><p>If we request 30GB per executor, AM will actually get 30GB for executor + 7% of 30 GB = ~ 33GB</p><p>Too much memory results in excessive garbage collection delays. Running tiny executor like single core and just enough memory throws away the benefits of parallelism in the same JVM excessive replications.</p><p>There are 2 ways to configure the executor and cores:</p><p><strong>Static Allocation</strong>: The values once set does not change. The values are given as a part of spark-submit.</p><p>Consider an example:</p><p>we have a cluster with 4 nodes, each node have 4 cores and 32GB memory.</p><p><strong>Tiny Executor:</strong> Tiny executor is 1 executor/core.</p><p>num-executor = 4 * 4 =16, executor-core = 1, executor-memory = 32/4 = 8GB</p><p>With tiny cluster the replication is multiplied and sometimes becomes large enough to give OOM errors. We are also missing the advantage of running the parallel task for the same JVM. Too much shuffling and network IO.</p><p>There is risk or running out of resources to compute a partition. Since each partition cannot be computed on more than one executor, the size of each partition is bounded by the space they have to be computed. Thus, we risk running into memory problems, or spilling to disk if we need to shuffle, cache unbalanced data, or perform very expensive narrow transformations.</p><p>Too many executors may not be an efficient use of our resources. Each executor has some overhead, and there is some cost to communicating between executors even if they are on the same node. Minimum memory overhead is just under 400 MB. Thus if we have many 1 GB executors, nearly 25 percent of the space that each executor will use on our cluster has to be used for overhead rather than computation.</p><p><strong>Big Executor</strong>: Big executor means 1 executor/node.</p><p>num-executor = 4 * 1=4, executor-core = 4, executor-memory = 32/1 = 32GB</p><p>With Big Executor we are not able to take the full advantage running more parallel jobs. Very large executors may cause delays in garbage collection, since a larger heap will delay the time until a GC event is triggered and consequently GC pauses may be larger.</p><p>Many cores per executor seems to lead to poor performance, due to some limitations from HDFS on handling many concurrent threads.</p><p><strong>Balanced Executor</strong>: we have 4 core/node. leave 1 core/node for the Hadoop daemon as we have 4 cores this leaves us to 3. hence 3 cores/node available. We can here take executor-cores=3.</p><p>3 * 4 = 12 these are the total cores available. Hence the number of executor is = 12/3 = 4 leaving 1 executor for the application master hence num-executor=3.</p><p>number of executor/node = (3/4) . memory/executor = 32*(3/4) = 24</p><p>counting the memory overhead = 7 % 24 ~ 3GB</p><p>executor=memory = (24–3) 21GB</p><p>Hence num-executor = 3, executor-core = 3, executor-memory = 21GB</p><p>This is how a parameters are set to achieve best of both world(Tiny and Big executor).</p><p>Remember more the number of cores in the same JVM is good for parallelism and less Network IO, less Garbage collection, less replication. Less the executor the better it is as this gives less Network IO, this means less shuffling and less time to execution.</p><p><strong>Dynamic Allocation: </strong>The values of the executors change dynamically as the number of task and load increases/decreases in the spark job. This helps to use just enough resource for the job and free up or add resource on need basis.</p><p>Upper bound for the number of executors if dynamic allocation is enabled is infinity. So this says that spark application can eat away all the resources if needed. In a cluster where we have other applications running and they also need cores to run the tasks, we need to make sure that we assign the cores at cluster level.</p><p>There are few parameter that needs to be set when using the dynamic allocation.</p><p>spark.dynamicAllocation.enabled= true/false =&gt; true specify that we would be using dynamic allocation of the executor in the spark job.</p><p>Other parameters follow.</p><p>spark.dynamicAllocation.initialExecutors</p><p>spark.dynamicAllocation.minExecutors</p><p>spark.dynamicAllocation.maxExecutors</p><p>When running Spark on YARN, each Spark executor runs as a YARN container. Apart from the setting of the YARN container we discussed above there are few setting in YARN what govern the minimum and maximum size of the executor. Check this interesting answer at <a href="https://stackoverflow.com/questions/38331502/spark-on-yarn-resource-manager-relation-between-yarn-containers-and-spark-execu">StackOverflow</a>.</p><h3>How to predict/control cost?</h3><p>A spark cluster can be created on cloud or on-prem. With the advent of the cloud most of the big data cluster and services are moving to cloud due to the obvious reasons.</p><p>It is now preferred to have the Big Data cluster on the cloud. On-prem you have numerous, uncontrolled expenses like hardware failure, unexpected absence, changing regulations, employee attrition. With cloud you pay for the service you use and asset you used. It’s the labor not the service. Select the right level of service, reduce the service cost by shortening the compute time.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_lt-fFcCFgvg_8_DkUtxsg.png" /></figure><p>Refer above chart for the how the cost is being calculated for. This is not the exact cost that comes in but will give you an idea of how these values are calculated and help you choose the best option for your job.</p><p>Apart from the compute cost there comes the developer cost we need to add in order to get true cost. developer cost can be taken as 50$ an hour and this can be multiplied with Actual Duration from above chart in order to get the result.</p><p>Developer Cost = 50 $ * ( Actual Duration)</p><h3><strong>VM Selection</strong>:</h3><p>Consider below chart for reference. We can see more the number of cores and VMs the more will be we the shuffling. sometimes this can be advantageous but most of the it might give an issue. All shuffle data must be written to disk and then transferred over the network. Each time that you generate a shuffling shall be generated a new stage. All shuffle data must be written to disk and then transferred over the network. Each time that you generate a shuffling shall be generated a new stage.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*j2FijfrGnNKPOnltGXYZ5w.png" /></figure><h3><strong>VM Category</strong>:</h3><p><strong>Memory Optimized: </strong>They are designed to give the faster performance for workloads that process large data in memory. They are best suited for ML workloads with data caching. We have job with lots of iterations on the job.</p><p><strong>Compute Optimized</strong>: They are ideal for your most performance-intensive workloads. They are ideal for ETL jobs with full file scan and no data reuse, Structured Streaming jobs.</p><p><strong>Storage Optimized</strong>: Storage optimized instances are designed for workloads that require high, sequential read and write access to very large data sets on local storage. They are optimized to deliver tens of thousands of low-latency, random I/O operations per second (IOPS) to applications. They are best suited for ML and DL workload with data caching, Data Analytics and Analysis, Spark Caching is the requirement.</p><p><strong>GPU Optimized</strong>: GPU instance for most deep learning purposes. Training new models is faster on a GPU instance than a CPU instance. You can scale sub-linearly when you have multi-GPU instances or if you use distributed training across many instances with GPUs. AL/ML workloads with very high workloads( large memory and large compute).</p><h3>Guide for Calculation of no. of VMs required</h3><ol><li>Calculate the data size on the disk. Assuming the dataset is of 100GB.</li><li>maxPartitionByte configuration in spark is set to 128MB by default.</li><li>calculate the partition it would create. 100GB = 102400MB/128MB = 800 partitions</li><li>Decide the category of VM that would be required. These are based on SLA, Quota, budget. refer above for the different category. for calculation purpose consider VM of 64GB, 36 Cores</li><li>Select the number of iterations you are ok with. for now consider you want the job to complete in 2 iterations.</li><li>Number of VMs = 800(Partitions) / (32 (cores) * 2 (iterations)) = 12 VMs</li></ol><p>These are just a rough calculations. The values differ based of use cases. the best way to find out is to calculates these values, test, adjust around these values and use the one is most suitable for your job.</p><h3>Other Criterion for Spark Cluster</h3><p>There are other criterion which are as important as above but we tend to ignore these. The criterion includes who would be using this cluster, what will the cluster used for, where would the data reside, when are the results needed. lets try to answer these.</p><h4>Who will be using the cluster?</h4><p>There are different kinds of people who can use these Spark Cluster. There are Data Analyst &amp; SQL Analyst who use the cluster to generate reports and browse through different datasets to analyze and generate report, There are Data Scientist who functions like Data Analyst but from the exploratory point of view or they can use the cluster to train different machine learning models do advanced analytics, There are Data Engineer who are preparing the data which would be used by the down stream systems/people for their purpose and then there are everyone else like Manager who use it for dashboard purpose.</p><h4>Data Analyst/SQL Analyst:</h4><p>They pull the data aggregate it and generate report. They are the guy would be working on(eg: silver or gold) data. They work on Highly repetitive queries since they are analyzing. They run it, change the parameter, run it again etc. there is a lot of back and forth they are doing during the analyzing phase. When doing so there might be a large gap between the queries. When doing analysis they are quick to join datasets to get the required results. Regularly employs caching for faster execution of queries.</p><h4>Data Scientist:</h4><p>Data Scientist classically functions like a Data Analyst. Since the approach for Data Scientist is more scientific, more statistical in that they are more inclined to use heavy compute. They regularly engage in data preparation to train ML Models with highly iterative job.</p><h4>Data Engineer:</h4><p>Data Engineer have the tendency to produce end to end applications. In that they are setting up a pipeline like there is some raw data that is coming out of some corporate system that into a bronze gets landed in a data lake expose it then augment it and curate it to silver and gold tables so that the lakehouse is consumable by everyone. They are driven by SLAs a lot more than data scientists and data analysts where they have a larger exploratory function. Data engineers are looking at the scenarios where this has to be done by a particular timeline. Scenarios like data comes in at midnight and report has be generated by morning before work starts or has to be near real time. They also support for Data Analyst, SQL Analyst, Data scientist we also do the things like productionizing jobs, take ML models and produce reports.</p><p>Their job is non repetitive queries, scheduled jobs. We have a point A to point B process and we are going to run it and it might be scheduled.</p><p>Data Engineer does the heavy job w.r.t the data received there demand for cluster would be the highest among the others discussed.</p><p>Depending on who is using the cluster we can then size the cluster, use the machines required, decide on the budget etc.</p><h4>What will the Cluster used for?</h4><p>Apart from who is using the cluster it is also important to know what would that cluster be used for. This helps us when creating the cluster to know how many VMs to request for, how much memory the nodes should have etc. There are different kind of job which Spark support to name few Ad-hoc Analysis, Streaming Jobs, Report generation, ETL/ELT, Training ML and Deep learning models, Data pipeline etc.</p><p>Every job has some different requirement it not recommended to use the cluster used for ETL/ELT jobs for Report generation since ETL/ETL uses heavy cluster to data loading and transformation as compared for report generation.</p><p>Streaming jobs use the cluster where VMs are compute optimized as compared to the Data pipeline where VMs are required to be storage/memory optimized in order to process the data.</p><h4>Where will the Cluster and Data reside?</h4><p>This question is w.r.t the cost of hosting the cluster. Where we would host the cluster which is most cost efficient. Hosting cluster on the cloud is on the trend for the reasons discussed above it is advantageous in terms of cost and usage. They are infinitely scalable — use as much as you need and pay as you use. Hardware and Software concerns are offloaded to the cloud provider who are experts in their respective domain.</p><h4>When are the SLA?</h4><p>These refer to the how long it takes to deliver data like whether you want the result in real time, near real time or scheduled time frame.</p><p>Depending on the SLA we need to create the cluster so that job completes in the time frame decided, it is fault tolerant, it can handle unprecedented request that comes in etc.</p><p>The cluster creation is a very important task in Big Data processing job. These takes some of your efforts in the initial stage but will help you for your long run for your job. The most important thing is to adjust, experiment and retest until you find your right combination of SLA, Cost, Performance for your job.</p><p>The list seems never ending, nevertheless I listed the one I knew. Hope this helps.</p><p>Thanks for reading!!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cf0cb9f33f1c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Persist, Cache and Checkpoint in Apache Spark]]></title>
            <link>https://medium.com/@badwaik.ojas/persist-cache-and-checkpoint-in-apache-spark-ae71783ce3dd?source=rss-6b750069b897------2</link>
            <guid isPermaLink="false">https://medium.com/p/ae71783ce3dd</guid>
            <category><![CDATA[persisting]]></category>
            <category><![CDATA[checkpointing]]></category>
            <category><![CDATA[apache-spark]]></category>
            <category><![CDATA[caching]]></category>
            <category><![CDATA[optimization]]></category>
            <dc:creator><![CDATA[Badwaik Ojas]]></dc:creator>
            <pubDate>Mon, 10 Apr 2023 05:55:29 GMT</pubDate>
            <atom:updated>2023-04-10T05:55:29.577Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/proxy/0*lKRdBi0tHAn7odir.png" /></figure><h3>Cache and Persist</h3><p>What a Data Analyst/Data Scientist do? <strong>Data Analyst</strong> pull the report, aggregate it and generate report. They work on highly repetitive queries since they are analyzing. They run it, change the parameter, run it again etc. there is a lot of back and forth they are doing during the analyzing phase. <strong>Data Scientist</strong> classically functions like a Data Analyst. Since the approach for Data Scientist is more scientific, more statistical in that they are more inclined to use heavy compute as oppose who randomly exploring data.</p><p>The datasets that Data Analyst/Scientist use is generally big in size. In case of Spark whenever we query the data it goes from the initial stage of reading the file from source and generating the results. Querying it once is ok imagine querying it repeatedly this would sometimes be frustrating.</p><p>Consider ETL/ELT jobs moving data around is expensive, both in time and money. In practice, your work will be connected to a central data store and the queries you run will pull data from there. In general, each time you run a query, you are going all the way back to the original store to fetch the data.</p><p>All of this overhead can be avoided using Cache or Persist.</p><p>One of the significant parts of Apache Spark is the ability to store things in memory during the computation. This is a neat trick that you can use as a way to speed up access to commonly queried tables or pieces of data. This is also <strong>great for iterative algorithms</strong> that work over and over again on the same data. While many see this as a panacea to speed issue but other concepts like data partitioning, clustering, bucketing can end up having a greater effect on the execution of the job than caching. However, remember these are all tools in your tool kits.</p><p>Though Spark provides computation 100 x times faster than traditional Map Reduce jobs, If you have not designed the jobs to reuse the repeating computations you will see degrade in performance when you are dealing with billions or trillions of data. Hence, we may need to look at the stages and use optimization techniques as one of the ways to improve performance.</p><p>In an application that reuse the same datasets over and over, one of the most useful optimization is caching and persisting. Caching is to place a Dataframe or Table into temporary storage across the executors in your cluster and make subsequent reads faster.</p><h4>Cache:</h4><p>Cache() method is used to cache the intermediate results of the transformation so that other transformation runs on top of cached Dataframe will perform faster. Caching the result of the transformation is one of the optimization tricks to improve the performance of the long-running Spark applications/jobs.</p><pre>var df = spark.read.schema(emp_schema).csv(&quot;\path\to\emp\emp.csv&quot;)<br>df.cache()<br>df = df.withColumn(&quot;increment&quot;, lit(1000))<br>df.show()</pre><h4>What happens when we cache?</h4><p>After calling the cache() function nothing happens with the data but the query plan is updated by the Cache Manager by adding a new operator. Actual Caching happens when you call Spark Actions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/914/1*kS4XI13Zq8tlKWrgmI1Wyw.png" /></figure><p>It works on the concept of lazy evaluation i.e. until and unless we call for an action nothing is materialized. When we call for the action its starts by reading the file and it sees it is cached, the data is then cached in the memory and next it does some transformation and then perform the action.</p><p>You see that in case of the second transformation it loads the data from the Cache and does not go to the source to fetch the data.</p><p>Spark automatically monitors cache usage on each node and drops out old data partitions in a least-recently-used (LRU) fashion. So least recently used will be removed first from cache.</p><p>Both Caching and Persisting are used to save the Spark RDD, Dataframe, and Dataset’s. But, the difference is, RDD cache() method default saves it to memory (MEMORY_AND_DISK) whereas persist() method is used to store it to the user-defined storage level.</p><h4>Persist</h4><p>Persist is similar to Cache the only difference is it can take argument and that too is optional. If no argument is given which by default saves it to MEMORY_AND_DISK storage level and the second signature which takes StorageLevel as an argument to store it to different storage levels.</p><p>Please refer this <a href="https://spark.apache.org/docs/3.3.2/rdd-programming-guide.html#rdd-persistence">Spark Documentation</a> for details of different StorageLevel.</p><p>To explicitly remove the Cache data from the memory we use an API unpersist.</p><pre>df.unpersist()</pre><p>In case of Caching and Persisting the lineage is kept intact which means they are fault tolerant and meaning if any partition of a Dataset is lost, it will automatically be recomputed using the original transformations that created it.</p><p>As you work with data in Spark, you will often want to reuse a certain dataset. It is important to be careful how you use caching, because it is an expensive operation in itself. If you are using the dataset once, for example the cost of pulling and caching is greater than that of the original data pulling. Once the data is cached. Catalyst Optimizer will only reach back to the location where the data was Cached.</p><p>Aside from those above use cases, you should not cache dataframe because it is likely that you’ll degrade the performance of your application. Caching consumes cluster resources that could otherwise be used for task execution. Caching prevents spark from performing query optimization.</p><p>The abuse of cache feature can sometime lead to more performance problems. It gets in the way of the Catalyst Optimizer, cripples predicate pushdown. they don’t understand the amount of time it takes to populate the Cache vs the read operation that is pulling from the Cache.</p><p>With great powers come great responsibilities :)</p><h4>Checkpoint</h4><p>Checkpointing is the act of saving an RDD to disk so that future references to this RDD point to those intermediate partitions on disk rather than recomputing the RDD from its original source. This is similar to caching except that it’s not stored in memory, only disk.</p><p>Consider the following code. Step 1 is setting the Checkpoint Directory. Step 2 is creating a employee Dataframe. Step 3 in creating a department Dataframe. Step 4 is joining of the employee and department Dataframe then ckeckpointing the join_df.</p><pre>spark.sparkContext.setCheckpointDir(&quot;\path\to\checkpoint\dir&quot;) // Step 1<br>var emp_df = spark.read.schema(emp_schema).csv(&quot;\path\to\emp\emp.csv&quot;) //Step 2<br>var dept_df = spark.read.schema(dept_schema)<br>                   .csv(&quot;\path\to\dept\dept.csv&quot;) //Step 3<br>var join_df = emp_df.join(dept_df, Seq(&quot;deptid&quot;), &quot;left&quot;) // Step 4<br>join_df = join_df.checkpoint() // Step 5<br>join_df.show() // Step 6</pre><p>Now lets see the Spark UI for the difference between <strong>with checkpoint</strong> and <strong>without checkpoint</strong>.</p><p><strong>Without Checkpoint</strong>: You see only one job is created. The Logical Plan is the complete plan that is required for achieving the target goal. The DAG as shown below has the complete flow where the data is loaded in two Dataframe and a join is made between the Dataframe.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/834/1*x5WHqf8DMxWhK-bGV6_U9Q.png" /><figcaption><strong>Without Checkpoint</strong></figcaption></figure><p><strong>With Checkpoint</strong>: You see a separate job is created when a checkpoint is called. This function materializes the RDD and stores into the checkpoint directory we have specified in the code. Once the Dataframe is checkpointed the subsequent reads would be from the Checkpoint location.</p><p>You see the Logical plan here it is truncated when compared to the Logical plan for the above.</p><p>DAG is also different in this case. Here we can see the starting point becomes the point where it was checkpointed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/689/1*bHc9UML-8o_djChw0bC5pQ.png" /><figcaption>With Checkpoint</figcaption></figure><p>Checkpointing is used to truncate the logical plan of this Dataset, which is especially useful in iterative algorithms where the plan may grow exponentially.</p><p><strong>When to Checkpoint?</strong> Every time a computed partition needs to be cached, it is cached into memory. However, Checkpoint does not follow the same principle. Instead, it waits until the end of a job, and launches another job to finish Checkpoint. An RDD which needs to be checkpointed will be computed twice; thus it is suggested to do a rdd.cache() before rdd.checkpoint(). In this case, the second job will not recompute the RDD. Instead, it will just read cache.</p><p>Persist / Cache keeps lineage intact while checkpoint breaks lineage. lineage is preserved even if data is fetched from the cache. It means that data can be recomputed from scratch if some partitions of in Cache are lost. In the second case lineage is completely lost after the checkpoint and it doesn’t carry an information required to rebuild it anymore.</p><h4>What is the difference between cache and checkpoint ?</h4><p>Here is the an answer from Tathagata Das:<br>There is a significant difference between cache and checkpoint. Cache materializes the RDD and keeps it in memory (and/or disk). But the lineage (computing chain) of RDD (that is, seq of operations that generated the RDD) will be remembered, so that if there are node failures and parts of the cached RDDs are lost, they can be regenerated. However, checkpoint saves the RDD to an HDFS file and actually forgets the lineage completely. This allows long lineages to be truncated and the data to be saved reliably in HDFS, which is naturally fault tolerant by replication.</p><p>Furthermore, rdd.persist(StorageLevel.DISK_ONLY) is also different from checkpoint. Through the former can persist RDD partitions to disk, the partitions are managed by blockManager. Once driver program finishes, which means the thread where CoarseGrainedExecutorBackend lies in stops, blockManager will stop, the RDD cached to disk will be dropped (local files used by blockManager will be deleted). But checkpoint will persist RDD to HDFS or local directory. If not removed manually, they will always be on disk, so they can be used by the next driver program.</p><p>Points to remember here is when Hadoop MapReduce executes a job, it keeps persisting data (writing to HDFS) at the end of every task and every job. When executing a task, it keeps swapping between memory and disk, back and forth. The problem of Hadoop is that task needs to be re-executed if any error occurs, e.g. shuffle stopped by errors will have only half of the data persisted on disk, and then the persisted data will be recomputed for the next run of shuffle. Spark’s advantage is that, when error occurs, the next run will read data from checkpoint, but the downside is that checkpoint needs to execute the job twice.</p><p>Caching is extremely useful than checkpointing when you have lot of available memory to store your RDD or Dataframes if they are massive.</p><p>Caching will maintain the result of your transformations so that those transformations will not have to be recomputed again when additional transformations is applied on RDD or Dataframe, when you apply Caching Spark stores history of transformations applied and re compute them in case of insufficient memory, but when you apply checkpointing spark throws away all of your transformations and stores finally Dataframe into HDFS forever.</p><p>The main problem of checkpointing is to store the data into HDFS which is slower than caching. you also need to setup checkpointing location on HDFS. persist(StorageLevel.DISK_ONLY) also has does similar thing but it stores history of your transformations. Checkpointing is mainly used in stateful transformation that combine data across multiple batches. In such transformations, the generated RDDs depend on RDDs of previous batches, which causes the length of the dependency chain to keep increasing with time and to avoid such unbounded increases in recovery time.</p><p>With this we come to an end to this Blog. Choose well and use wisely.</p><p>Thanks for Reading!!</p><h4>References:</h4><ol><li><a href="https://mallikarjuna_g.gitbooks.io/sparkinternals/content/cache-and-checkpoint.html">Spark Internals</a></li><li><a href="https://community.databricks.com/s/question/0D53f00001GHVhrCAH/when-to-use-cache-vs-checkpoint">DataBricks Community</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ae71783ce3dd" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Health check using Spark UI]]></title>
            <link>https://medium.com/@badwaik.ojas/health-check-using-spark-ui-b0d3d26e08a0?source=rss-6b750069b897------2</link>
            <guid isPermaLink="false">https://medium.com/p/b0d3d26e08a0</guid>
            <category><![CDATA[apache-spark]]></category>
            <category><![CDATA[monitoring]]></category>
            <category><![CDATA[performance]]></category>
            <category><![CDATA[spark-ui]]></category>
            <category><![CDATA[tuning]]></category>
            <dc:creator><![CDATA[Badwaik Ojas]]></dc:creator>
            <pubDate>Tue, 04 Apr 2023 18:17:01 GMT</pubDate>
            <atom:updated>2023-04-04T18:17:01.056Z</atom:updated>
            <content:encoded><![CDATA[<h3>Spark Job Health check using Spark UI</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*lKRdBi0tHAn7odir.png" /></figure><p>When a person goes to a doctor for the health related issues, doctors in order to know the root cause might ask for some laboratory test or scans. Using the reports of the scans or tests they now have much details in order to provide you the solution.</p><p>Likewise there are many cases we face during our big data job where we know that the data isn’t that big but why does it takes so long time to run a simple query, why are we getting OOM errors, want to monitor task distribution in the job, why the job is hanged etc. In order to get to the root cause of the issue we lookout for the Spark UI. Apache Spark provides a suite of web user interfaces that you can use to monitor the status and resource consumption of your Spark cluster.</p><p>Spark UI for Data Engineers are the go to as the scans and blood test are for the doctors they help in diagnosis for the issue. Using Spark UI you can get to the root cause of the issue your job is facing. Spark UI is a very important monitoring tool for your Spark Job. Lets go through details for Spark UI in this Blog!!</p><p>When a spark job in initiated it starts the Spark UI it prints out the URI using which you would be able to access the UI. You can copy and paste the URI to access from any web browser. When you run locally this would be like <a href="http://localhost:4040/">http://localhost:4040</a></p><p>Consider the below code for analysis</p><pre>//Created Employee Dataframe<br>var emp_df = spark.read.schema(emp_schema).csv(&quot;/path/to/emp/emp.csv&quot;)<br>//Created Department Dataframe<br>var dept_df = spark.read.schema(dept_schema).csv(&quot;/path/to/dept/dept.csv&quot;)<br><br>//Joined the dataframe on department id<br>val join_df = emp_df.join(dept_df, Seq(&quot;deptid&quot;), &quot;left&quot;)<br><br>//Persist the joined dataframe<br>join_df.persist(StorageLevel.MEMORY_AND_DISK)<br><br>//Performed a collect action<br>join_df.collect()</pre><p>Lets open the Spark UI link to check on the execution. When you open the link you would see below tabs.</p><p>Pssssst: Get a knack of what a Spark Job looks like and its components from my <a href="https://medium.com/@badwaik.ojas/component-of-spark-application-job-1a52b3ea2bc">blog</a> for better understanding.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/704/1*XHJ5OSEsTFgrVLZW0CNBfQ.png" /></figure><h4>Jobs</h4><p>The Jobs tab displays a all jobs in the Spark application and once you click on any of the job it takes you to the details page for that particular job. The Jobs tab is like a summary page where you can see all the job executed for the current Spark Application run.</p><p>This page also has details like:</p><ol><li>User: Current User</li><li>Total Uptime</li><li>Scheduling Mode: Refer <a href="https://spark.apache.org/docs/3.3.2/job-scheduling.html">spark documentation</a> for details.</li><li>Number of jobs with status: Active, Completed, Failed.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/184/1*5O5yCIYOAmXrgDppDVCA3Q.png" /></figure><p>Below this you would see the Event timeline. Event timeline tells you about the event that occurred in a timely manner about the spark application. Refer below SS(Screen Shot) for details.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0KQ8bNlB-wQie2nIPmGFlg.png" /></figure><p>Here we can see the timeline for the job we executed earlier. It tells right from the starting of the job about the executors being added and once executor was added then we started with the job execution starting from first job to the last job in order.</p><p>Then There is table for all the Active job that are being executed below this Event timeline in my case since its a very small job I was not able to capture that.</p><p>Below the Active Job we see the table were there is a list of the Jobs that is Completed. The Table contains details for JobID, Description for the Job, Time it was submitted, Duration, Stages it went through in order to complete the job and status of the success stage vs total number of stages. Refer below SS for the job we executed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QIUM7o753lY1umlMzqSpjw.png" /></figure><p>From the spark code we executed looking at the above SS we can conclude it was divided in 2 jobs. For every Job, Description column has a link using which go to the details page for that particular job.</p><p>In the details page we have the status for the job we have opened, its status and event timeline we just saw.</p><p>There is a DAG Visualization available, looking at this we would be we get a know the job execution. Refer SS below</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/888/1*3GsyoYqEPjoc82pgKXYj3g.png" /></figure><p>Looking at the DAG we can see, the job is divided into 3 stages. In our case.</p><ol><li>Stage 0: Creating the First(emp_df) Dataframe</li><li>Stage 1: Creating the Second(dept_df) Dataframe</li><li>Stage 2: Executing the join and creating Third(join_df) Dataframe</li></ol><p>Lastly in the details page you would see the details of all stages executed for the job. Refer SS below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CcrbAQeV2E7QOOWf8SV7iQ.png" /></figure><p>If you have multiple jobs all will be displayed on the Jobs page in Spark UI and along with the duration and If you are analyzing a job having performance issue. We can start by checking the jobs in this tab and look for the one having issue, go into the depths to the task level to see where exactly is the root cause and help tune the job.</p><h4>Stage</h4><p>A Job is divided into one or more Stages depending on the complexity. There is a Stage tab after Job which is the consolidation of all the Stages in the Spark Application. Its like a summary page we had for Job.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gaYflkagDG50qaXwxss__g.png" /></figure><p>Clicking on any one the stage will lead you to the page where it will have all the details w.r.t the Stage.</p><p>Stage is further divided into tasks. Depending on the number of task we have for that stage we would have that displayed on the Stage details page. Just like the Job details page had the Stage level details. The Stage details has the Task level details.</p><p>In our instance consider the Stage 0. in our example context the Stage details page has details like the total time to read the dept_df. There is also a visual representation of the directed acyclic graph (DAG) of this stage, where vertices represent the RDDs or DataFrames and the edges represent an operation to be applied.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/369/1*3YKo0_lEAKks5Zs5gSFG6Q.png" /></figure><p>Summary metrics for all task are represented in a table and in a timeline. The Timeline event are helpful in order to get us a of data distribution along the time, compute time among tasks and other useful parameter. apart form the metrics you see below there are two other metrics that is Spill (Disk) and Spill(Memory) these metrics are available only if we have Spill in our Stage else you won’t see these fields.</p><p>These Metrics are helpful if you are looking for any performance issues in you job. These helps to find any Skewness, Spill, Serialization related problems in our Job if any. If we find them there are always Optimization techniques we can use in order to rectify them.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KYRol3AoP9NPyERR7tnVAA.png" /></figure><p>Below these metrics we have Aggregated Metrics by Executor. These shows the task distribution among the executors, the time it took to complete and data distribution.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0zy7ZUKhtJ3IPqcYXpsrNw.png" /></figure><p>In the last we have the all the task level details. It has the entry for each and every task that executed on executor level in the current stage. The task is lowest level of job execution part and it work directly on the data. This tell us about the task information like time it took to complete, data size it worked/ number of records etc. This the ground level where we can detect the task level issue related to data distribution.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MFuknFwnCu1slrR7uKJ-tw.png" /></figure><h4><strong>Storage</strong></h4><p>The Storage tab displays the persisted RDDs and DataFrames, if any, in the application. The summary page shows the storage levels, sizes and partitions of all RDDs, and the details page shows the sizes and using executors for all partitions in an RDD or DataFrame.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sHCcQXw5nghCXmHR6Sy15A.png" /></figure><p>You can click the RDD name for obtaining the details of data persistence, such as the data distribution on the cluster.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CtKXFj3KJbC_-m7-RZkB7w.png" /></figure><h4>Environment</h4><p>The Environment tab displays the values for the different environment and configuration variables, including JVM, Spark, and system properties.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BfcMneNcUfCmy9RvIpBsWw.png" /></figure><h4>Executor</h4><p>The Executors tab displays summary information about the executors that were created for the application, including memory and disk usage and task and shuffle information. The Storage Memory column shows the amount of memory used and reserved for caching data.</p><p>The Executors tab provides not only resource information (amount of memory, disk, and cores used by each executor) but also performance information (GC time and shuffle information).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xsI9YfPKFQuyvIM1UWGjpA.png" /></figure><h4>SQL Tab</h4><p>If the application executes Spark SQL queries, the SQL tab displays information, such as the duration, jobs, and physical and logical plans for the queries. See below SS. This has information of the queries being executed, time it took to execution and the job ids that needed for the successful execution. You can click on the job id to view for the details.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YZHZrPuPdf4JnNIuhT505w.png" /></figure><p>If we click the ‘collect at Demo.scala:124’ link of the last query, we will see the DAG and details of the query execution.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/201/1*dBPoQwFqU23cJpegMHPhtw.png" /></figure><p>Clicking the ‘Details’ link on the bottom displays the logical plans and the physical plan, which illustrate how Spark parses, analyzes, optimizes and performs the query. Steps in the physical plan subject to whole stage code generation optimization, are prefixed by a star followed by the code generation id. This helps to understand what is that being executed on the ground and help tune the query if needed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*523NqQza91XhGoKwm-AEuw.png" /></figure><p>With this we come to an end to the Spark UI Blog. As mentioned this a very import part for analysis for your big data job in order to make the them performant and efficient. Helps to make great choice in creating cluster, detect skewness, spills, task distribution help make optimization on your job in order to run them smoothly.</p><p>I can say that any data engineer you are able to read Spark UI efficiently could be the best in this field. Hope you liked it.</p><p>Thanks for reading!!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b0d3d26e08a0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Dissecting Spark-Submit!!!]]></title>
            <link>https://medium.com/@badwaik.ojas/dissecting-spark-submit-701ab1712231?source=rss-6b750069b897------2</link>
            <guid isPermaLink="false">https://medium.com/p/701ab1712231</guid>
            <category><![CDATA[apache-spark]]></category>
            <category><![CDATA[spark-submit]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[deployment]]></category>
            <category><![CDATA[cluster-manager]]></category>
            <dc:creator><![CDATA[Badwaik Ojas]]></dc:creator>
            <pubDate>Sun, 02 Apr 2023 18:19:23 GMT</pubDate>
            <atom:updated>2023-04-02T18:19:23.979Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*yMbrIyvlPtSQOcfi.png" /></figure><p>For development purpose we use either spark-shell or an IDE to develop and test our Big Data application. Once everything looks good we want to promote the application to run production loads for that there is a process for submitting standalone applications to Spark called <strong>spark-submit</strong>, were you can submit a precompiled application to Spark.</p><p>Spark makes it easy to develop and create big data programs. Spark also makes it easy to turn your interactive exploration into production applications with spark-submit, a built-in command-line tool. spark-submit does one thing: it lets you send your application code to a cluster and launch it to execute there. Upon submission, the application will run until it exits (completes the task) or encounters an error. You can do this with all of Spark’s support cluster managers including Standalone, Mesos, and YARN.</p><p><strong>spark-submit</strong> offers several controls with which you can specify the resources your application needs as well as how it should be run and its command-line arguments. Lets look at the basic structure of <strong>spark-submit</strong>.</p><h4>Spark-Submit Skeleton</h4><p>Below is what a spark-submit skeleton looks like.</p><pre>spark-submit \<br>--class &lt;main-class&gt; \<br>--master &lt;master-url&gt; \<br>--deploy-mode &lt;deploy-mode&gt; \<br>--jars &lt;external-dependent-jars-requireed-by-application&gt;<br>--conf &lt;key&gt;=&lt;value&gt; \<br>&lt;application-jar-or-script&gt; \<br>[application-arguments]</pre><p>Lets discuss each variables used in the spark-submit in brief.</p><p><strong>class: </strong>This is the class name that we want to execute from the jar file we have deployed</p><p><strong>master: </strong>This represents the cluster manager we are going to use for our deployment. This could be one of Standalone, Mesos, Yarn or Local.</p><p><strong>deploy-mode</strong>: This represents how the spark application is deployed in. It could be client or cluster. (default -&gt;client)</p><p><strong>conf</strong>: If there are spark configuration you want to set for your application. You can use this using this variable. Example: — conf spark.sql.shuffle.partitions=10</p><p><strong>jars</strong>: If there are any dependent jars that are required by your application that needs to added here.</p><p><strong>application-jar-or-script</strong>: This is the Spark Application jar along with path that has our actual code.</p><p><strong>py-files</strong>: In case we are running spark application developed in python script we added dependent python files like .zip, .egg, or .py files these needs to be added here.</p><p>likewise there are many <strong>configuration/customization</strong> that can be added. use <strong>spark-submit — help</strong> to watch for different options available.</p><pre>// Sample spark-submit command for word count <br>spark-submit \<br>--class org.demo.spark.examples.WordCount \ // -&gt; Class name to execute<br>--master local \ // -&gt; This could be local, yarn, spark://hostname:port, mesos://hostname:port<br>--deploy-mode client \ // -&gt; this could be client or cluster<br>--executor-memory 20G \ // -&gt; executor memory for the spark application<br>--total-executor-cores 100 \ // -&gt; executor core for the spark application<br>/path/to/jar/wordcount.jar \ // -&gt; jar having Spark Application code<br>1000 // -&gt; argument if any needed for the Spark-Application to run</pre><h4>Cluster Managers</h4><p>In master variable in spark-submit we need to specify the cluster manager we would be using. Unless you are using a high-level managed service, you will have to decide on the cluster manager to use for Spark. Spark supports three aforementioned cluster managers: Standalone clusters, Hadoop YARN, and Mesos. lets discuss these in brief.</p><p><strong>Standalone</strong>: Spark’s standalone cluster manager is a lightweight platform built specifically for Apache Spark workloads. Using it, you can run multiple Spark Applications on the same cluster. It also provides simple interfaces for doing so but can scale to large Spark workloads. The main disadvantage of the standalone mode is that it’s more limited than the other cluster managers — in particular, your cluster can only run Spark. It’s probably the best starting point if you just want to quickly get Spark running on a cluster, however, and you do not have experience using YARN or Mesos.</p><p>Starting a standalone cluster requires provisioning the machines for doing so. That means starting them up, ensuring that they can talk to one another over the network, and getting the version of Spark you would like to run on those sets of machines. After that, there are two ways to start the cluster: by hand or using built-in launch scripts.</p><p>Start the master process on the machine that we want that to run on, using the following command: $SPARK_HOME/sbin/start-master.sh</p><p>Once the master is up we would get a Master URI something like spark://Host:Port. We then use this URI in the worker nodes to start the slave machine using the command. $SPARK_HOME/sbin/start-slave.sh &lt;master-spark-URI&gt;. As soon as you’ve run that on another machine, you have a Spark cluster running! This process is naturally a bit manual.</p><p>You can then use the Master URI we saw above in the spark-submit command to execute spark application in Standalone mode. Standalone mode has different configuration that can be used, refer this <a href="https://spark.apache.org/docs/3.3.2/spark-standalone.html">Spark Documentation on standalone mode</a> where they has detailed about different configuration and properties about this cluster manager.</p><p>Standalone mode have Static partitioning.</p><p>PS: The partition here does not refer to the partition of RDD at all. Here it refers like in YARN and Mesos you can start with like 10 executors and increase it to like 20 executors or decrease it to 5 executors it is all when the program is running. In standalone mode at the start you will mention that you require x number of cores or like 10 executors you are then stuck with this till the end of application. You cannot increase or decrease during this mode.</p><p><strong>YARN</strong>: I have already written a Blog regarding <a href="https://medium.com/@badwaik.ojas/yarn-the-hadoop-os-4ca882e0335c">YARN</a> but here I will give a brief about this cluster manager. Hadoop YARN is a framework for job scheduling and cluster resource management.</p><p>When submitting applications to YARN, the core difference from other deployments is that — master will become <strong>yarn</strong> as opposed the master node IP, as it is in standalone mode. Instead, Spark will find the YARN configuration files using the environment variable <strong>HADOOP_CONF_DIR</strong> or <strong>YARN_CONF_DIR</strong>. Once you have set those environment variables to your Hadoop installation’s configuration directory, you can just run spark-submit. These variables needs to be set in $SPARK_HOME/spark-env.sh.</p><p>You can run your Spark jobs on Hadoop YARN by specifying the master as YARN in the spark-submit command-line arguments.</p><p>Like standalone mode yarn also has different configuration and properties that can be set based on the requirement refer this <a href="https://spark.apache.org/docs/3.3.2/running-on-yarn.html">Spark Documentation on Yarn</a> for more information.</p><p>YARN has dynamic partitioning.</p><p><strong>Mesos</strong>: Apache Mesos is another clustering system that Spark can run on. Mesos intends to be a datacenter scale-cluster manager that manages not just short-lived applications like Spark, but long-running applications like web applications or other resource interfaces. Mesos is the heaviest-weight cluster manager, simply because you might choose this cluster manager only if your organization already has a large-scale deployment of Mesos. This deployment mode is not widely used for the reasons mentioned above due to this I won’t go in much details. If you are looking for details on mesos will refer you to the <a href="https://spark.apache.org/docs/3.3.2/running-on-mesos.html">Spark Documentation on Mesos</a>.</p><p>Submitting cluster mode applications is fairly straightforward and follows the same spark-submit. In the master variable we need to set the Master URI as we did for standalone mode but here in this case it would be like mesos://Host:Port.</p><p>Mesos has dynamic partitioning.</p><p>This all about cluster managers in Spark in short. Based on the requirement you can choose the cluster manager that suits your need.</p><h4>Deployment Modes</h4><p>An deployment mode gives you the power to determine where the resources i.e. driver, executor and cluster manager are physically located when you go to run your application. There are 3 deployment mode available in Spark. Local, Client and Cluster.</p><p><strong>Cluster</strong>: Cluster mode is probably the most common way of running Spark Applications. In cluster mode, a user submits a pre-compiled JAR, Python script, or R script to a cluster manager. The cluster manager then launches the driver process on a worker node inside the cluster, in addition to the executor processes. This means that the cluster manager is responsible for maintaining all Spark Application–related processes.</p><p><strong>Client</strong>: Client mode is nearly the same as cluster mode except that the Spark driver remains on the client machine that submitted the application. This means that the client machine is responsible for maintaining the Spark driver process, and the cluster manager maintains the executor processes. we are running the Spark Application from a machine that is not co-located on the cluster. These machines are commonly referred to as gateway machines or edge nodes.</p><p><strong>Local</strong>: Local mode is a significant departure from the previous two modes: it runs the entire Spark Application on a single machine. It achieves parallelism through threads on that single machine. This is a common way to learn Spark, to test your applications, or experiment iteratively with local development.</p><p>Enough for the deployment to get you started and differentiate to help you choose. now lets get to the actual part.</p><h4>What happens when we execute spark-submit command?</h4><p>Assuming cluster is already running with one Master and three Worker node and we deploying the application using Cluster Mode. The first step is to submit the Spark Application now from the local machine we would Submit a request to the Cluster Manager Master node and initially we would explicitly ask for resource Spark Driver only if the request is accepted by the Cluster Manager it starts the driver process in one of the worker node.</p><p>The client process that submitted the original job exists and the application is off and running on the cluster.</p><p>Now that the driver process has been placed on the cluster, it begins running user code. This code must include a SparkSession that initializes a Spark cluster (e.g., driver + executors). The SparkSession will subsequently communicate with the cluster manager asking it to launch Spark executor processes across the cluster. The number of executors and their relevant configurations are set by the user via the command-line arguments in the original spark-submit call.</p><p>The cluster manager responds by launching the executor processes and sends the relevant information about their locations to the driver process. After everything is hooked up correctly, we have a “Spark Cluster”. Spark goes about its way executing code.</p><p>The driver and the workers communicate among themselves, executing code and moving data around. The driver schedules tasks onto each worker, and each worker responds with the status of those tasks and success or failure.</p><p>After a Spark Application completes, the driver process exits with either success or failure. The cluster manager then shuts down the executors in that Spark cluster for the driver. At this point, you can see the success or failure of the Spark Application by asking the cluster manager for this information.</p><p>In case of the client mode deployment. The driver would we the local machine we submitted our job through. The driver would request for the executor node directly and those are allocated it would start executing the spark code and send all the relevant application to the client machine. In this case our machine would in direct contact with executor and cluster manager. If due to any reason we have issue in our local machine the driver would be terminated and job will fail.</p><p>This is all about the Spark-Submit you want to know in brief. Try running the command and experimenting with different variable and configuration and check their reaction on the job.</p><p>Hope you liked the job. Thanks for reading!!!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=701ab1712231" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>