<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Daniel Freitas on Medium]]></title>
        <description><![CDATA[Stories by Daniel Freitas on Medium]]></description>
        <link>https://medium.com/@danielrsfreitas?source=rss-3a7b31df382f------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*CbOTg5bEz_WRNTZC3OYLIA.jpeg</url>
            <title>Stories by Daniel Freitas on Medium</title>
            <link>https://medium.com/@danielrsfreitas?source=rss-3a7b31df382f------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 10:57:42 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@danielrsfreitas/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Review — Udacity Data Architect Nanodegree]]></title>
            <link>https://medium.com/@danielrsfreitas/review-udacity-data-architect-nanodegree-d6de81ebe54d?source=rss-3a7b31df382f------2</link>
            <guid isPermaLink="false">https://medium.com/p/d6de81ebe54d</guid>
            <category><![CDATA[data-architecture]]></category>
            <category><![CDATA[nanodegree]]></category>
            <category><![CDATA[udacity]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[review]]></category>
            <dc:creator><![CDATA[Daniel Freitas]]></dc:creator>
            <pubDate>Tue, 27 Dec 2022 14:51:53 GMT</pubDate>
            <atom:updated>2022-12-27T14:51:53.167Z</atom:updated>
            <content:encoded><![CDATA[<h3>Review — Udacity Data Architect Nanodegree</h3><p><strong>If you are into the creative part of designing data systems, perhaps this course is for you!</strong></p><h3>Introduction</h3><p>The data and analytics industry is here to stay. According to the <a href="https://www.marketresearchfuture.com/reports/data-analytics-market-1689">Market Research Future</a>, it expect to reach USD 306 billion by the end of 2030.</p><p>It doesn’t matter if the trend is on Big Data, Data Virtualization, Data Fabric or AI. It is always about data. Corporations should always leverage data to make decisions, and this is when things get dangerous. Bad data is often worse than no data.</p><p>Every data has its own particularities. Understanding how to fit the pieces on the puzzle is key for the success of every company. There should be a professional who understands the business problem and holds the knowledge of technical solutions.</p><p>That is when the Data Architect comes to play and that is what the Udacity Data Architect ND aims to prepare you for.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*oQhLdwizx1enXEfo" /><figcaption>Photo by <a href="https://unsplash.com/@dawson2406?utm_source=medium&amp;utm_medium=referral">Stephen Dawson</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>Target Audience</h3><p>The Udacity Data Architect Nanodegree program is intended for people who want to improve their data-related skills and translate business requirements into technical tasks for building enterprise data management systems.</p><p>Based on my experience, to enjoy the content entirely, I believe the student should have an established technical background to manipulate basic programming logic and SQL statements.</p><p>The pre-requisites are:</p><ul><li>Basic familiarity with traditional batch data processing and traditional service architecture</li><li>Intermediate Python</li><li>Intermediate SQL</li><li>Operating systems, including UNIX, Linux, and MS Windows</li><li>Basic familiarity with ETL/Data Pipelines</li></ul><h3>Nanodegree Content</h3><p>The program curriculum breaks down into 04 chapters. Each one with its unique instructor, subject and a grading project. By the end of the ND you will have developed a total of 04 projects.</p><p>I will drill down through each of them below.</p><h4>Data Architecture Foundations</h4><p><a href="https://github.com/danielrsfreitas/data_architect/tree/main/data_architecture_foundations"><em>My personal GitHub — Data Architecture Foundations</em></a></p><p>This first class talks about beginners concepts of Data Architecture and their importances. The idea is to give a first glance of what will be deeply explained in the upcoming courses.</p><p>This chapter’s lessons cover concepts like:</p><ul><li>Basics of Data Architecture</li><li>Difference between Business and Functional Requirements</li><li>Model and Schema</li><li>Normalization (3NF)</li><li>Entity Relationship Diagram (ERD)</li><li>DDL and DML</li></ul><p>Udacity offers a challenge at the end of each chapter. This way you can move forward step by step consistently, like a &quot;checkpoint&quot; to make sure you have understood the most important concepts.</p><p>This one introduces the design and build of a database for an HR department. To achieve that, the student has on hands a few documents: IT best practices, a Memo written by the head of the HR department and business requirements. Those will drive him towards the final goal.</p><p>The deliverables are:</p><ul><li>A Design Document Explaining Architectural Decisions</li><li>SQL commands for DDL, DML and CRUD</li><li>Conceptual, Logic and Physical ERD</li></ul><p><strong>Personal opinion about this chapter</strong></p><p>This chapter and challenge delivers basic but very useful concepts. It accomplishes pretty well the mission of introducing the student to what he should expect from the entire course. It does that by showing beginners concepts from what is a stakeholder, what is a business requirement till what is normalization, ERD and DDL.</p><p>The final project can be tricky when it comes to deriving an ERD based on a plain text business requirements. It reminds me moments of my career that I had to see &quot;between the lines&quot; to understand what the client wanted to achieve.</p><h4>Designing Data Systems</h4><p><a href="https://github.com/danielrsfreitas/data_architect/tree/main/designing_data_systems"><em>My personal GitHub — Designing Data Systems</em></a></p><p>In the second chapter of this program, the lessons will introduce a data platform tool: you will get to know <a href="https://www.snowflake.com/en/">Snowflake</a> framework.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*4XBmITzSRKuU29wr" /><figcaption>Photo by <a href="https://unsplash.com/@aaronburden?utm_source=medium&amp;utm_medium=referral">Aaron Burden</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>The topics for this section are:</p><ul><li>Business Stakeholders</li><li>Data Artifacts (Data Dictionary, Data Flow, Data Stewardship, Dimensional Model)</li><li><a href="https://en.wikipedia.org/wiki/Online_transaction_processing">OLTP</a> vs. <a href="https://en.wikipedia.org/wiki/Online_analytical_processing">OLAP</a></li><li>Data Steps — Staging, Operation Data Store (ODS), Data Warehouse</li><li>Star Schema and Snowflake Schema</li></ul><p>The grading project is a design of a Data Warehouse for Reporting and OLAP using Snowflake. In this challenge the student will get to do something very relevant to a data architect routine, which is defining different sources, such as but not limited to, <a href="https://www.yelp.com/">YELP</a> and Climate Data on Snowflake. With that, given the business context stated in the project description, the student has to create all the steps from data raw source all the way to reporting on Snowflake as shown in the figure below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rTiZq2zA7Mwvd2w69bEnNg.png" /><figcaption>Data Flow on the final project of Designing Data Systems course</figcaption></figure><p>Along with that, it is also required the design of the data architecture in terms of OLTP for the ODS and OLAP (star schema) for the Data Warehouse.</p><p><strong>Personal opinion about this chapter</strong></p><p>This is the most thorough and useful course in the entire ND. Below you can find a list of benefits I got by going through the challenge:</p><ol><li>Uses Snowflake on the exercises and final project, which is a widely used tool on industry. Along with that, the course is mostly hands-on so I ended up feeling comfortable using the Snowflake tool and I can consider that I learned at least a good amount of functionalities of it.</li><li>The main concepts of the course are industry standards and broadly used in real world cases.</li><li>The final project uses real data to drive the required insights. It is a challenging project and demands understanding the chapter&#39;s concepts to be able to pass. The datasets also have good size that is not so big to make it too hard but is also not so small to act like a sample data.</li></ol><p>A small drawbacks:</p><ul><li>Duplicate content. Some content is already covered on the previous course, such as Logical Model, Conceptual Models. Some other will be covered in the Data Governance section, which makes more sense because they are related to Security, Data Ownership and so on.</li></ul><h4>Big Data Systems</h4><p><a href="https://github.com/danielrsfreitas/data_architect/tree/main/big_data_systems"><em>My personal GitHub — Big Data Systems</em></a></p><p>In this section, the goal is to introduce the Big Data world. It talks about ingestion, massive volume data process and storage.</p><p>One of the concerns when enrolling in such class is how to manage different resources and infrastructure assets, and Udacity has a great way to help with it: they provide an <a href="https://aws.amazon.com/">AWS</a> account you can use to run some products, such as EMR or EC2. It is worth mentioning that this account is limited and the user must pay attention on turning the cloud instance off when the work is done.</p><p>The main topics are:</p><ul><li>4V&#39;s — Volume, Velocity, Variety and Veracity</li><li>Horizontal Scaling vs. Vertical Scaling</li><li>Row oriented vs. Column oriented</li><li>Big Data Formats — JSON, CSV, ORC, AVRO, Parquet</li><li>Ingestion — Kafka and Sqoop</li><li>Storage — HDFS</li><li>Processing — Pig, Hive, MapReduce, Spark, YARN, Hbase</li><li>Security — Ranger, Knox, Atlas</li><li>OPS- Airflow, Zookeeper, Ambari</li><li>NoSQL — CAP Theorem, ACID</li><li>Change Data Capture — Apache Hudi</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*uodLivPzoF2YpPlx.png" /><figcaption>CAP Theorem</figcaption></figure><p>The grading project is a design of a Data Lake system for a Medical Data Processing Company. This time it is slightly different from the past ones. Now, it requests 4 deliverables:</p><ol><li>Data Lake Architecture</li><li>Data Lake Design Document</li><li>Data Lake Executive Presentation (slide deck)</li><li>Video Recording Session of myself presenting the slides</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*t_Ej2-hqtsOulLyk.png" /><figcaption>Final Project: Data Lake Architecture Diagram</figcaption></figure><p><strong>Personal opinion about this chapter</strong></p><p>The Big Data Systems course is the heaviest one in terms of exercise. It requires a lot of time if you want to go through every exercise proposed.</p><p>A good thing about this course is it embraces the open-source tools. Yet, it does not emphasises the tools that are already deprecated such as Sqoop. It talks about it as it was a right alternative for data ingestion.</p><h4>Data Governance</h4><p><a href="https://github.com/danielrsfreitas/data_architect/tree/main/data_governance"><em>My personal GitHub — Data Governance</em></a></p><p>Last but not least! Data Governance is the closing subject of the Data Architect Nanodegree.</p><p>This chapter covers:</p><ul><li>Metadata Management</li><li>Data Quality Management— Data Profiling, Data Quality Dimensions, Remediation and Monitoring</li><li>Master Data Management — Golden Record, Master Data Hub</li></ul><p>Imagine you are a new hire Data Architect of the SneakerPark Online Shoe Reseller. The grading project explores precisely that! You will be given some technical and business requirements. With that, you need to come up with:</p><ul><li>Enterprise Conceptual Model</li><li>Documenting the Metadata in an spreadsheet</li><li>Profiling a PostgreSQL database to find Data Quality issues and manage them</li><li>Mockup a Data Quality dashboard</li><li>Master Data Management Architecture</li><li>Detailed explanation of every decision made along the project</li></ul><p><strong>Personal opinion about this chapter</strong></p><p>Data Governance is a key aspect to turn the data of a company professional and organized. It is an important matter that sometimes is neglected because it doesn&#39;t offer short-term benefits.</p><p>Said that, the course is relevant in terms of up to date material, the instructor has a lot of experience. In addition, the order of the topics make sense because they are complementary.</p><p>This was the easiest grading project. Although it takes more time than the others, it is not a cognitive challenge when compared to the other chapters.</p><h3>Time spent</h3><p>Even though the course is intended for 4 months of study, 10h/week, it took me less than 3 months to finish it.</p><p>I am a slow learner. I like to go over all the exercises, take a moment to reflect and avoid rushing just to get the certificate. I usually read the extra supporting material and that also increases the amount of time I take to finish these kinds of self-pacing courses.</p><h3>Is it worth it?</h3><p>It is finally time to wrap up and answer — is it worth paying money on the Data Architecture Udacity Nanodegree?</p><p>For professionals with hands-on experience, the program won&#39;t offer much of a challenge because it focuses more on translating business requirements into technical tasks.</p><p>The DA ND is for people who want to creatively put the technical pieces together with a common goal of delivering value to the stakeholders. In other words, the idea is to provide a background for those who want to be the interface between business and technical.</p><p>If you see yourself in a position of improving your skills towards that direction, I say this course is worth for you.</p><h3>GitHub</h3><p>Checkout my personal GitHub with the results for each step of the DA ND.</p><p><a href="https://github.com/danielrsfreitas/data_architect/tree/main/data_governance">https://github.com/danielrsfreitas/data_architect/</a></p><h3>References</h3><ul><li><a href="https://www.udacity.com/course/data-architect-nanodegree--nd038">How to Become a Data Architect | Udacity</a></li><li><a href="https://github.com/danielrsfreitas/data_architect">GitHub - danielrsfreitas/data_architect</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d6de81ebe54d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Use case: Updating Apache Spark major version of an enterprise data platform]]></title>
            <link>https://medium.com/ibm-data-ai/use-case-updating-apache-spark-major-version-of-an-enterprise-data-platform-8610ad77d3ab?source=rss-3a7b31df382f------2</link>
            <guid isPermaLink="false">https://medium.com/p/8610ad77d3ab</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[upgrade]]></category>
            <category><![CDATA[spark]]></category>
            <dc:creator><![CDATA[Daniel Freitas]]></dc:creator>
            <pubDate>Wed, 26 Oct 2022 00:09:09 GMT</pubDate>
            <atom:updated>2022-10-26T00:09:09.644Z</atom:updated>
            <content:encoded><![CDATA[<p>Imagine a big tech company.</p><p>This company has hundreds of thousands of employees. Some employees work in the Finance sector, others work in the Marketing, some others are from HR, and so on. Each of these areas make business decisions on a regular basis. To support such decisions they use their data segregated for their purpose, which works so well, right? But now, let’s add a little bit more complexity to that scenario: consider a director of a certain business unit. He or she wants to know how much revenue for the second quarter of 2022 came from the engineering contracts, grouped by region and industry sector. Ok, now things are getting more interesting. Where should he or she seek for that information? Perhaps different sources of data can lead to different results. That is misleading and dangerous. Maybe, the data to serve that conclusion cannot be aggregated because they are in different standards.</p><p>Solution: establish a queryable single source of truth.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*bpvK-RR3taVYZSn9" /><figcaption>Photo by <a href="https://unsplash.com/ja/@markuswinkler?utm_source=medium&amp;utm_medium=referral">Markus Winkler</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><blockquote>Right! But I’m here for the technical part, c’mon!</blockquote><p>I hear you! Let’s move to the good part!</p><p>The long paragraph above was to present the business context that the platform I will talk about aims to solve.</p><p>Let’s call this solution as Enterprise Data Platform (EDP). The EDP stack is:</p><p>1. Python 3</p><p>2. <a href="https://spark.apache.org/docs/2.4.2/running-on-kubernetes.html">Apache Spark on Kubernetes</a> 2.4.2</p><p>3. Airflow 2.2.1</p><p>4. Kubernetes 1.23</p><p>5. IBM Cloud Products (<a href="https://www.ibm.com/cloud/object-storage">Cloud Object Storage</a>, <a href="https://www.ibm.com/cloud/db2-warehouse-on-cloud">Db2</a>, <a href="https://www.ibm.com/cloud/cloudant">Cloudant</a>, …)</p><p>EDP runs different data pipelines. More than 100 flows for different purposes. From different sources, all the way to a data lake and then to a data warehouse organized in star-schema.</p><p>Let me give you some numbers to better visualize the platform size. A daily overview on the resource allocations can be retrieved from the Cloud Monitoring Service, <a href="https://www.ibm.com/pt-en/cloud/sysdig">Sysdig</a>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*L9j1fGfwA5PriEdSZ8Ay-w.png" /><figcaption>Daily overview of the EPD’s resource allocation</figcaption></figure><p><strong>5 TiB</strong> of memory allocation and <strong>2.17K</strong> cores of cpu used by the pods in the Kubernetes cluster.</p><p>When narrowing to the main data pipeline that runs in the product, it takes:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/395/1*HeFK2ybLvr0rf8QiCjNOug.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*i9I8pSJfzaLV_byArJvDgQ.png" /><figcaption>Sysdig resource allocation metrics</figcaption></figure><h3>Apache Spark</h3><p><a href="https://spark.apache.org/">Apache Spark</a> is a distributed data framework that makes our data pipeline faster. It is the core of our data pipeline engine. Thus, major changes on Apache Spark is critical as it presents a major risk of breaking the entire production operation.</p><p>Since EDP’s inception, the official Apache Spark version was a custom image based off of the 2.4.2 version. All the data ingestion pipelines were using Spark 2.4.2 taking advantage of its features.</p><h3>Why Migrate To Spark 3?</h3><p>The migration is critical because we have to ensure no existing pipeline will break when switching to the newer version.</p><p>The major topics when it comes to the decision about the version upgrade were:</p><ol><li>Security</li><li>Bug fixes</li><li>Community support</li><li>New features (Adaptive Query Execution, Dynamic Partition Pruning, etc)</li></ol><p>In the next paragraph I will talk about some of the hurdles I had to overcome to succeed on this migration.</p><h3>Test Plan</h3><p>The first challenge that came up was to try building the whole EDP system on top of the raw Spark 3 image. A naive approach showed to be a complete disaster, as expected. Attempts to compile the code to make at least one single successful build failed miserably.</p><p>EDP uses a Python 3.7-stretch with Debian image as base. On top of it is the <em>SparkBase </em>image, which installs some packages dependencies, such as Java and a proprietary framework.</p><p>After all the stack of images builds, it is time to get to the actual tests to verify if any breakage appears.</p><h4>Unit Tests</h4><p>The EDP is widely tested with over 522 unit tests and 56% of code coverage at a certain point. Thus, the first hurdle to overcome was the unit tests. All the suit should pass like they did for Spark 2.</p><p>The issue encountered relates to the Spark version chosen. The higher release version at the time was 3.2.1, even though the latest was 3.1.3, given the minor patch applied. Those versions were too different from each other.</p><p>It happens that, in 3.2.1, the unit test out of nowhere gets stuck and the issue was not 100% reproducible. Other than that, the stuck point was never the same, it could vary, making it really hard to predict. The Spark UI also didn’t bring any insight since the Spark session was up but the next task would never start. The adopted hypothesis was a possible issue with <a href="https://docs.pytest.org/en/7.1.x/">Pytest </a>when running with the Spark version 3.2.1. (*<em>Spoiler*: Initial tests show that spark 3.3.0 does not hang as well).</em></p><p>The solution was to try other Spark distributions to try reproducing the issue. It turns out that it does not. On Spark version 3.1.3 the Pytest does not hang.</p><p>A few adjustments made to have the tests passing:</p><ul><li>Switch the SQL Context into SparkSession</li><li>Adapt the binary string return on some DataFrame columns</li><li>Remove <em>sql_context.tableNames() </em>of the tests</li></ul><p>The first decision happens here. We have selected the version 3.1.3 to keep going and proceed with the next set of tests.</p><h4>Integration Tests</h4><p>Along with the unit tests, EDP also provides a set of integration tests. The integration tests checks every core component of the solution. It does so by creating test cases that isolate each one of those.</p><p>The test consists of:</p><p>1. Input data</p><p>2. Expected data</p><p>3. YAML file that describes the pipeline execution plan with the single operation to be tested</p><p>For example, a data join between two input sources. After every new change in the core, the test suit runs and verify if the new change introduced a point of failure.</p><p>Given that context, it was essential having the integration test passing after the Spark migration. It is also important to mention how useful the Integration Test Suit has proven to be on this migration. Without it, the Spark version upgrade would have been a lot more painful. Imagine running pipelines with real data to illustrate all the different kind of scenarios of data manipulation. It would be roughly unfeasible.</p><p>Getting back to the story. It turns out that the integration tests failed at first. They failed because the new version of Spark does not accept applying <em>na.fill</em> operation into columns that are not present in the DataFrame. Moreover, the <em>naFillColumns</em> was mandatory on one of the most used transformations. The only way out was to make the naFillColumns an optional parameter. Along with that, create a wrapper on the <em>na.fill</em> called <em>safe_fill_na. </em>This way, it first verifies if the column exist and then apply the filter.</p><p>The change to make integration tests pass was:</p><ul><li>Create a safe fill.na that only applies to columns that exists</li></ul><p>After all 157 tests were green, it was then time to move to the tests on the staging environment of the cluster.</p><h4>AB Tests</h4><p>When proceeding to the cluster tests, we should avoid the abrupt switching versions. This way, a comparison between versions could be done. The AB Tests approach was then developed to enable running two different versions of the EDP engine on staging, taking turns. We also didn’t want to test all pipelines and we thought that selecting only some would be biased.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/725/1*45qMcZQg-zfxQgDpQf8aIg.png" /></figure><p>When moving to AB tests on staging, one new issue appeared. This one related to <a href="https://issues.apache.org/jira/browse/SPARK-31404">SPARK-31404</a>. The issue occurs when trying to extract a dateType that is too old and was ingested using Spark 2 but now it is being extracted by Spark 3.</p><p>The first attempt to solve it was by setting the flag:</p><pre>spark.sql.legacy.parquet.datetimeRebaseModeInRead: LEGACY</pre><p>After doing that, many pipelines started to work. Though, it didn’t resolve for all.</p><p>Some runs were failing in the <em>ApplySchema</em> step. The <em>ApplySchema</em> is a step where EDP enforces the schema of the DataFrame. This step makes sure that the data will look like as expected in the data warehouse.</p><p>We noticed that even though the flag was set, the value was not affecting the <em>ApplySchema</em> operation when transforming the DataFrame into an RDD. Then, we decided to build a custom Spark 3.1.3 image forcing the <em>datetimeRebaseMode</em> to be always Legacy. This is acceptable for our use case. <a href="https://github.com/danielrsfreitas/spark/commit/bfaff8c66517a07f78cb5cab05cd17732943bbed">Here</a> is the change reference that we made on top of the v3.1.3 tag.</p><p>With that, all the images went through adjustments and rebuild before going back to test.</p><p>The pipelines that were failing now begun to pass and we could move on with more testing.</p><h4>Regression Tests</h4><p>The test engineers know that a robust unit test suit is not enough. Along with the UT, a well written integration test suit decrease the amount of errors in the code base. But, there are some situations that only pop up when a Regression Test or a Stress Test is executed. There are some edge cases that only appears when pushing the system to the limit.</p><p>With that in mind, we built a temporary infrastructure to run those Regression Tests. For that we:</p><ul><li>Deployed a dedicated Airflow instance</li><li>Created a Regression Test DAG where we included the most important data pipelines</li><li>Ran those pipelines using a custom code built on top of our latest Spark 3 version</li></ul><p>Those tests were important in many ways. First, we have confirmed that the most important DAGs were covered by this new Spark version.</p><p>Second, we encountered another specific issue.</p><p>The issue relates to the Dynamic Hive Partitioning feature that we use to load data into the Data Lake in a smart way. It was tricky to reproduce the issue. It shows up after a manual movement of folders in the data lake. For example, when someone moves parquet files across different folders using a cloud storage browser. During the next data ingestion the dynamic partitioning fails because it can’t rename hidden files generated in this movement.</p><p>To solve that, we chose to educate users to avoid manual manipulations of files in the data lake. Everything works fine if the user replaces the manual move by the CLI.</p><h3>Minor Issues Along The Way</h3><p>Small issues happened along the way and it is worth mentioning them even though they got fixed right away.</p><h4>AWS and Hadoop JAR</h4><p>The Hadoop version was also upgraded when moving to the Spark 3 version. We went from the Hadoop 2.7 to the Hadoop 3.3.1, which is also a major update. It happens that, to build the Spark 3.1.3 with Hadoop 3.3.1, the directive of the version has to set explicitly.</p><pre>./dev/make-distribution.sh — name custom-spark — pip — tgz -Phive -Phive-thriftserver -Dhadoop.version=3.3.1 -Pkubernetes</pre><p>Spark 3.1.3 with Hadoop 3.3.1 Build command</p><p>However, the build doesn’t bring the Hadoop 3 dependencies JAR files that we need on EDP. Due to that, the JAR were also brought to the image. Those are:</p><ul><li>aws-java-sdk-bundle-1.11.901.jar</li><li>hadoop-aws-3.3.1.jar</li><li>wildfly-openssl-1.0.7.Final.jar</li></ul><h3>Results</h3><p>When we first came up with the idea of upgrading to Spark 3, we were looking for bug fixes, new features and security improvements. However, we also wanted to verify improvements in two areas:</p><ol><li>Runtime performance</li><li>Stuck Pods Resolution</li></ol><p>For number 1, we were not expecting any improvements on this first phase. We were not enabling performance improvement features, which are the breakthrough changes of the Spark 3. Also, they get enabled by default on Spark 3.2.1 and on. We decided to not turn it on now and first observe the Spark 3 raw in production for some time. Only some pipelines have the Adaptive Query Execution enabled for experimenting.</p><p>About number 2, there is an issue that happens on Spark on Kubernetes where Spark is not able to finish itself. So even though the container execution reaches the end, the K8s pod remains in <em>Running</em> state. This is bad because our scheduler, the Airflow, relies on the pod state to take actions. The Spark 3 showed the same issue that Spark 2 already presented.</p><h4>Pipeline runs results</h4><p>In development, during a week, there is a total of <strong>733</strong> runs. Because of the AB Tests approach, we could compare between the versions. That was how we identified that no runtime performance degradation or improvement occurred.</p><p>After that, in one month, a total of <strong>9,712</strong> runs occurred in the EDP production and all of them were through Spark 3.1.3. The success rate of those is <strong>98,29%</strong> and none of the failures were related to the Spark version.</p><h4>Vulnerabilities</h4><p>To check the vulnerabilities in the image I have used the <a href="https://engine.anchore.io/">Anchore Engine</a>. The number of vulnerabilities in the image has decreased after the upgrade. The image has reached <strong>17%</strong> less vulnerabilities. <a href="https://nvd.nist.gov/vuln/detail/CVE-2022-26612">Here</a> is one example of vulnerability that was fixed.</p><h3>Conclusion</h3><p>The key takeaways of the upgrade process are:</p><h4>Test tooling matters</h4><p>A solid set of test tooling increase the confidence in the process. It is important having different kinds of tests. Unit, integration, smoke, regression, stress and so on. The more the better. This is a truth in software development in general. But, in data engineering, we have to be mindful that we handle data objects and those artifacts also need test. The data cannot get affected by a platform change.</p><p>Without a trustworthy data test platform the upgrade would be a pandora box. We would never know what data pipeline would brake if we didn’t test them all.</p><h4><strong>Stick to the plan</strong></h4><p>Sometimes it is frustrating and it seems that we have reached the end of the line. Where all possibilities went trough exhaustion, running out of ideas. One example of that was when the flag of <em>datetimeRebaseMode</em> was not reflecting the expected, even though that was the solution suggested by the Apache Spark Community. It is not an easy decision creating a custom image of a solid code base given current version didn’t fit our needs. We could have given up and said that “Our platform does not conform with the requirements”. But, in our minds, this would mean failure. How come only EDP is not able to upgrade? This was not acceptable.</p><p>It is important to have clarity on the procedures. In a complex problem you should always try to break it into smaller problems. Change one variable at a time to narrow down where the root cause of the issue may be. Establishing a clear procedure helps avoiding the feeling of “running in circles”.</p><h4><strong>Next steps</strong></h4><p>The next steps of this continuous works is the migration to the Spark 3.3.0, which released on 16th June 2022. The highlight of this release is the log4j version migration from 1 to 2 to account for the famous <a href="https://www.washingtonpost.com/technology/2021/12/20/log4j-hack-vulnerability-java/">security breach of log4j</a>. Other than that, performance improvements and feature enablements are also a path to follow after this present work.</p><h4>References:</h4><ul><li><a href="https://spark.apache.org/docs/latest/sql-migration-guide.html#upgrading-from-spark-sql-24-to-30">Migration Guide: SQL, Datasets and DataFrame</a></li><li><a href="https://issues.apache.org/jira/browse/SPARK-31404">file source backward compatibility after calendar switch</a></li><li><a href="https://wind010.hashnode.dev/optimizing-pytest-and-pysparkspark">Optimizing Pytest and Pyspark/Spark</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8610ad77d3ab" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ibm-data-ai/use-case-updating-apache-spark-major-version-of-an-enterprise-data-platform-8610ad77d3ab">Use case: Updating Apache Spark major version of an enterprise data platform</a> was originally published in <a href="https://medium.com/ibm-data-ai">IBM Data Science in Practice</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>