<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Robert Kossendey on Medium]]></title>
        <description><![CDATA[Stories by Robert Kossendey on Medium]]></description>
        <link>https://medium.com/@robert.kossendey?source=rss-91105643a921------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*2Tp5XAm9nVi124agcwGfYw.png</url>
            <title>Stories by Robert Kossendey on Medium</title>
            <link>https://medium.com/@robert.kossendey?source=rss-91105643a921------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 23:23:18 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@robert.kossendey/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Palantir Foundry — The Data Operating System That is Not Talked About Enough]]></title>
            <link>https://medium.com/better-programming/palantir-foundry-the-data-operating-system-that-is-not-talked-about-enough-9fb1c98a6b3d?source=rss-91105643a921------2</link>
            <guid isPermaLink="false">https://medium.com/p/9fb1c98a6b3d</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[palantir-foundry]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[palantir]]></category>
            <dc:creator><![CDATA[Robert Kossendey]]></dc:creator>
            <pubDate>Wed, 26 Apr 2023 19:35:26 GMT</pubDate>
            <atom:updated>2023-04-26T20:49:36.474Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*s0ghGolfg_RGx5GZVZ63lg.png" /></figure><h4><strong>Foundry</strong></h4><p>The world of data engineering is full of debates, with different schools of thought vying for supremacy. There are a lot of discussions around data modeling, e.g., Inmon vs. Kimball, but also around tooling, whether <a href="https://medium.com/@laurengreerbalik/how-fivetran-dbt-actually-fail-3a20083b2506">Fivetran + DBT are rent-seeking</a>, or the everlasting debate of Snowflake vs. Databricks. Today I want to shed some light on a tool I have rarely seen discussed in any LinkedIn posts, Medium articles, or HackerNews threads.</p><p>That tool is Palantir Foundry. Some might have heard of it already, but what exactly is Foundry? While Palantir is primarily known for its work with the government, they also have a commercial offering, Foundry the Data Operating System. First and foremost, it is designed to be a complete end-to-end data platform, a mesh of services containing everything you need to manage your data, from ingestion to transformation, storage, visualization, and Machine Learning. But that is only the tip of the iceberg.</p><p>Foundry takes a radically different approach to traditional cataloging and schema design, focusing on semantics and kinetics. You create an <a href="https://www.palantir.com/docs/foundry/ontology/overview/">Ontology</a>, defining objects, their properties, and the links between them. It connects your data to the real world, serving as a digital twin and as a shared language between all the stakeholders of an organization. This enables, e.g., business users to build applications on top of the Ontology to fully operationalize the data without thinking of data as data frames but as real-world objects.</p><p>The Ontology is such a groundbreaking concept that it could be the subject of an article on its own. However, in this article, I want to highlight three features that make Palantir Foundry stand out to me.</p><h4><strong>Disclaimer</strong></h4><p>Before we dive into the features of Palantir Foundry, it’s important to note that I have yet to personally use Foundry. My knowledge is based on conversations with users, videos, and documentation.</p><p>It’s also worth mentioning that there are valid reasons why Palantir Foundry is talked about less than other data platforms. For one, it’s not an off-the-shelf SaaS product with pay-per-use pricing. Instead, it requires expensive licenses that may be out of reach for some organizations. Additionally, Palantir hasn’t always been transparent about its product capabilities, although this is changing as they continue to improve its documentation and communication with users.</p><p>With that said, let’s take a closer look at three unique features of Palantir Foundry.</p><h3><strong>Treating Data Like Code</strong></h3><p>Software developers often utilize version control systems to coordinate their work on a codebase. This allows multiple engineers to safely contribute to the same codebase without interfering with each other’s work. Foundry approaches data in a similar way that software developers approach code. It enables many individuals to interact with the same data and make changes without disrupting others’ progress using branching. It allows you to diverge from the main path and work on data within your branch. After making desired changes, you can merge your branch back into the main branch once you are satisfied with the results.</p><p>Let us imagine you are working on a feature branch and want to commit a change to the main branch, e.g., rename a column in a Spark Job. You can use the Compare feature of Foundry to compare the output dataset on the feature branch to the dataset on your main branch. The comparison includes a lot of helpful dataset stats, like size, file count, and row count. You can even compare on column level, looking at the percentage of NULL values, Min/Max, or Mean. This allows you to ensure that your changes are purely metadata related in this use case.</p><p>If you are sure your changes are ready to be merged, you create a pull request showing you many helpful things. Firstly you can see, as you are used to, the files that have been changed by the commit. But now the magic begins! Foundry will provide an impact analysis showing you the affected datasets, their schema changes, and whether the affected datasets pass pre-defined health checks.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JQ5LyXuHrLkOWv6rSzj7XA.png" /><figcaption>Impact analysis</figcaption></figure><p>But there is more! Since Foundry captures the lineage of each dataset, there is a pipeline review for each pull request, visualizing the changes made to the pipeline.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tKvt0l3eIBCVlttlmhsmFQ.png" /><figcaption>Lineage review</figcaption></figure><p>Those features not only help people working on the pipelines but also people that are reviewing pull requests, helping them to make sure that no unexpected changes occur.</p><h4><strong>Typescript functions with pre-generated types</strong></h4><p>Foundry provides a service called Foundry Functions that enables users to interact with objects from the ontology via a TypeScript AWS Lambda-like function.</p><p>One of the critical benefits of Foundry Functions is that it generates interfaces for objects imported from the Ontology. This makes it easy to interact with those objects by accessing their properties, traversing links between them, or aggregating collections. Typical use cases are:</p><ul><li>Custom aggregations for dashboards.</li><li>The calculation of custom metrics.</li><li>Even complex edits to the Ontology itself.</li></ul><p>You can reuse functions across multiple services, such as Foundry Workshop, which allows users to build no-code applications.</p><p>Besides running unit tests automatically on commit, it also provides a live preview tab that allows users to test their functions in real time, making the development process iterative and easy.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/955/1*8J8UVkyoZI5rdUCj8kwkeg.png" /><figcaption>Preview Tab</figcaption></figure><p>Foundry Functions brings collaboration in an enterprise to a whole new level. It allows regular Software Engineers to work directly on the data provided by Data Engineers without thinking about columns and rows in a table.</p><h4><strong>Quiver’s Graph</strong></h4><p>Quiver is a service that provides advanced analytical and dashboarding capabilities on top of your Ontology. You can build dashboards that can be shared across your organization or even embedded in other services. It is similar to products like Tableau or PowerBI.</p><p>One feature that makes it stand out to me is its two modes.</p><p>First is the canvas mode, which is very familiar to people that have already used traditional BI tools, which provides an empty view you can fill with visuals. Then there is Graph Mode, where it’s getting interesting.</p><p>Let us imagine you are inheriting the ownership of a dashboard, and the creator has already left the company. The dashboard displays some numbers and KPIs, and you are tasked with changing some. Here is where Graph mode comes into play. It provides a dependency graph of your analyses, showing the lineage of every visual contained, allowing the user to understand why a specific visual shows data. I’ve certainly been in such a situation, and I can not stress enough how handy this feature would have been.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5njHkGwDBKfJHKo_etasOg.png" /><figcaption>Graph Mode showing the lineage of a dashboard</figcaption></figure><h4><strong>Conclusion</strong></h4><p>Foundry is a vast product, and there is lots and lots to talk about. I hope you enjoyed the article and found the features as exciting and unique as I do.</p><h4><em>Additional resources</em></h4><ul><li><a href="https://www.youtube.com/watch?v=XEZ-bTtW8Z4&amp;ab_channel=PalantirDevelopers">Code Repositories</a></li><li><a href="https://www.youtube.com/watch?v=v4vJgI_M3Jw&amp;ab_channel=PalantirDevelopers">Foundry Functions</a></li><li><a href="https://www.youtube.com/watch?v=lpHFL4xjVPg&amp;ab_channel=PalantirDevelopers">Quiver Graph Mode</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9fb1c98a6b3d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/better-programming/palantir-foundry-the-data-operating-system-that-is-not-talked-about-enough-9fb1c98a6b3d">Palantir Foundry — The Data Operating System That is Not Talked About Enough</a> was originally published in <a href="https://betterprogramming.pub/">Better Programming</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Lakehouse — A resumé]]></title>
            <link>https://medium.com/claimsforce/lakehouse-a-resum%C3%A9-750368e57914?source=rss-91105643a921------2</link>
            <guid isPermaLink="false">https://medium.com/p/750368e57914</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[delta-lake]]></category>
            <category><![CDATA[athena]]></category>
            <category><![CDATA[data-lakehouse]]></category>
            <dc:creator><![CDATA[Robert Kossendey]]></dc:creator>
            <pubDate>Sun, 19 Mar 2023 09:26:22 GMT</pubDate>
            <atom:updated>2023-03-19T09:26:22.098Z</atom:updated>
            <content:encoded><![CDATA[<h3>Lakehouse — A resumé</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Wu6hevN1LSYyrbNYwVfwPA.png" /></figure><h4>About this blog series</h4><p>At claimsforce, our initial approach to big data was a two-tier architecture consisting of a Data Lake stage in Amazon S3 and a Data Warehouse stage in Amazon Redshift (outline<a href="https://aws.amazon.com/blogs/startups/how-claimsforce-built-a-future-proof-lake-house-with-aws/"> here</a>). Over time we realized that having two stages comes with disadvantages like engineering and maintenance effort, infrastructure costs, and data staleness. We aim to replace the combination of a Data Lake and Data Warehouse with a unified system — the Lakehouse. In this blog series, we will document our journey toward a Lakehouse setup.</p><h4>Recapitulation</h4><p><a href="https://medium.com/claimsforce/lakehouse-the-journey-unifying-data-lake-and-data-warehouse-bef7629c143a">The first article</a> of this blog post series outlined the challenges we are facing with the traditional Two-Tier architecture. We discovered how the Lakehouse approach helps us overcome those challenges and came up with an idea for our future data infrastructure.<a href="https://medium.com/claimsforce/lakehouse-running-delta-lake-on-top-of-aws-glue-181133a916f3"> The second article</a> explained how we ran <strong>Delta Lake</strong> on top of AWS Glue and what caveats this had. After that, we concluded that we had to look for a new ETL tool and compared two promising alternatives, <strong>Databricks</strong> and <strong>AWS EMR</strong>. We describe this in the<a href="https://medium.com/claimsforce/lakehouse-databricks-vs-aws-emr-87e30c00b791"> third article</a>.</p><p>With Databricks as our tool of choice, we dive into the last piece of this series that covers the migration process and a cost comparison.</p><h4><strong>Migration to Databricks</strong></h4><p>Migrating ETL jobs can be dangerous. Dashboards, ML services, and, thus, end users depend on tables fed by the jobs. Since we were modifying our architecture at two places by changing our ETL tool and switching from Redshift to Athena, we needed to be especially careful to avoid breaking existing user workflows. To minimize risk, we started running the new Databricks jobs, which write to S3, parallel to our existing Glue jobs that wrote into Redshift. For quality assurance, we loaded data from both sources into Spark and used the great<a href="https://github.com/MrPowers/chispa"> Chispa</a> framework to make sure the DataFrames contained the same data.</p><p>To keep track of which files we have already processed, we used the Job Bookmark feature of Glue. As this is unavailable on Databricks, we had to look for an <strong>alternative mechanism</strong> allowing incremental processing. As it turns out, Spark has such a feature out of the box if you use the Streaming API.</p><p>One common <strong>misconception</strong> is that Spark Streaming automatically means “Always-on” expensive clusters. Instead, you can run incremental jobs using the <a href="https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#triggers">Trigger.Once feature</a>. Unfortunately, we were unaware of <a href="https://docs.databricks.com/ingestion/auto-loader/index.html">Databricks Auto Loader</a> back then, which does essentially the same but is more scalable and performant while potentially <strong>being cheaper</strong> due to the usage of native cloud APIs to get lists of files that exist in storage.</p><h4><strong>Integrating with Athena</strong></h4><p>Athena did not support working with Delta Lake tables natively back when we were migrating our infrastructure. That meant that Athena could not leverage features like data skipping and could <strong>not identify tombstone files</strong> that were no longer part of the table. To prevent Athena from reading stale data, you had to create a <strong>manifest file</strong> containing the list of files representing the table’s latest version. Therefore we wrote a helper function that updates the manifest file for a given table every time it gets updated.</p><p>At re:invent 2022, AWS<a href="https://aws.amazon.com/about-aws/whats-new/2022/12/athena-enhances-read-support-delta-lake-table-format/"> announced</a> that Athena can now natively read from Delta Lake tables. There are still some caveats, DML is not supported, and you need to register the table in the Glue catalog, but all in all, this is a <strong>vast improvement</strong> from before and a <strong>significant step</strong> in the right direction.</p><h4><strong>Outlook</strong></h4><p>Finally, let us look into the future at what we could improve. One big goal is migrating <strong>from Athena to Databricks SQL</strong> since they now support serverless querying. That would enable us to fully use the Unity Catalog, which provides sophisticated security features and <strong>automated lineage tracking</strong>.</p><p>Additionally, we want to <strong>contribute to open-source</strong> projects centered around Delta Lake, like<a href="https://github.com/MrPowers/mack"> Mack</a>,<a href="https://github.com/MrPowers/jodie"> Jodie</a>, or<a href="https://github.com/MrPowers/levi"> Levi</a>, since this migration made us <strong>appreciate</strong> open-source tech even more than before.</p><h4><strong>Cost Comparison</strong></h4><p>Now to the fun part, looking at the costs and evaluating whether the migration was <strong>successful</strong>. First, a look into the costs generated by AWS Glue vs. Databricks + EC2 costs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/962/0*LYt_3Qaz-yElWcjd" /></figure><p>As you can see, we reduced our costs by nearly a factor of <strong>8</strong>! That is attributed to the fact that Glue is just more <strong>expensive</strong> than Databricks and EC2 combined and that we can run multiple jobs on the <strong>same cluster</strong>, reducing costs for start-up times. We also used the migration to do some housekeeping, merging jobs and removing jobs that aren’t of use anymore. Furthermore, the<a href="https://docs.databricks.com/runtime/photon.html"> Photon Engine</a> could <strong>speed up complex jobs</strong> and lower total costs.</p><p>Looking at Redshift vs. Athena, we can see that our costs also decreased by 75%. Since our querying workload consists mainly of report queries running in a set interval instead of ad hoc analytics, we benefit tremendously from the switch to serverless.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/970/0*e3_vnZVFwcwlwd8p" /></figure><p><strong>Conclusion</strong></p><p>Overall, we are <strong>more than satisfied</strong> with the outcome of our <strong>Lakehouse</strong> migration.</p><p>We reduced our overall costs by <strong>80%</strong> while improving our developer experience drastically. We don’t have to maintain a Redshift cluster anymore. Instead, we store all the data in a <strong>single place</strong>, S3. Further, the core of our infrastructure is powered by <strong>open source</strong>, namely Apache Spark and Delta Lake. That empowers us to move away from Databricks if we are ever unhappy with the service.</p><p>I hope you enjoyed our short journey. If you’re interested, there is a <a href="https://www.youtube.com/watch?v=2tsvgM8ECVk&amp;ab_channel=DeltaLake">tech talk</a> with <strong>Denny Lee</strong> and me where we dig a little deeper into <strong>Delta Lake</strong>.</p><p><em>written by: </em><a href="https://www.linkedin.com/in/johannes-kotsch-28a874173"><em>Johannes Kotsch</em></a><em> &amp; </em><a href="https://www.linkedin.com/in/robert-kossendey-303b0019a"><em>Robert Kossendey</em></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=750368e57914" width="1" height="1" alt=""><hr><p><a href="https://medium.com/claimsforce/lakehouse-a-resum%C3%A9-750368e57914">Lakehouse — A resumé</a> was originally published in <a href="https://medium.com/claimsforce">claimsforce</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Lakehouse — Databricks vs. AWS EMR]]></title>
            <link>https://medium.com/claimsforce/lakehouse-databricks-vs-aws-emr-87e30c00b791?source=rss-91105643a921------2</link>
            <guid isPermaLink="false">https://medium.com/p/87e30c00b791</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[data-lakehouse]]></category>
            <category><![CDATA[aws-emr]]></category>
            <dc:creator><![CDATA[Robert Kossendey]]></dc:creator>
            <pubDate>Thu, 05 Jan 2023 20:09:16 GMT</pubDate>
            <atom:updated>2023-01-05T21:59:12.961Z</atom:updated>
            <content:encoded><![CDATA[<h3>Lakehouse — Databricks vs. AWS EMR</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oJm9EuWoGt2s9qPjEjVApQ@2x.png" /></figure><p><strong>Disclaimer</strong>: <em>The decision on which ETL tool to use took place in May, EMR Serverless was in preview, and it did not support Delta Lake natively back then.</em></p><h4>About this blog series</h4><p>At claimsforce, our initial approach to big data was a two-tier architecture consisting of a Data Lake stage in Amazon S3 and a Data Warehouse stage in Amazon Redshift (outline<a href="https://aws.amazon.com/blogs/startups/how-claimsforce-built-a-future-proof-lake-house-with-aws/"> here</a>). Over time we realized that having two stages comes with disadvantages like engineering and maintenance effort, infrastructure costs, and data staleness. We aim to replace the combination of a Data Lake and Data Warehouse with a unified system — the Lakehouse. In this blog series, we will document our journey toward a Lakehouse setup.</p><h4>Recapitulation</h4><p>The <a href="https://medium.com/claimsforce/lakehouse-the-journey-unifying-data-lake-and-data-warehouse-bef7629c143a">first article</a> of this blog post series outlined the challenges we are facing with the traditional Two-Tier architecture. We discovered how the Lakehouse approach helps us overcome those challenges and came up with an idea for our future data infrastructure. The <a href="https://medium.com/claimsforce/lakehouse-running-delta-lake-on-top-of-aws-glue-181133a916f3">second article</a> explained how we ran Delta Lake on top of AWS Glue and what caveats this had. After that, we concluded that we had to look for a new ETL tool.</p><h4>Finding the Perfect ETL Tool</h4><p>As part of our search for the best ETL tool for our data pipelines, we looked at several different options. The main requirement was that the ETL tool runs PySpark code since we wanted to keep our existing scripts the same upon this migration.</p><p>We started by compiling a list of potential tools and comparing their features and capabilities. Because of their <strong>integration with S3</strong>, <strong>developer experience</strong>, <strong>scalability</strong>, <strong>performance</strong>, and <strong>pricing</strong>, we narrowed our choices to a shortlist of two tools: Amazon EMR and Databricks. We then took a closer look at these two options, comparing them in different aspects against our current tool, AWS Glue. As evaluation criteria, we considered <strong>ease of use,</strong> <strong>features</strong>, <strong>vendor lock-in</strong>, and <strong>pricing</strong>. The following presents the results of the comparison.</p><h4><strong>Ease of use</strong></h4><p>Both solutions support <strong>more</strong> <strong>Spark versions</strong>, including more recent ones than Glue. That is useful if you want to take advantage of the latest features and improvements in Spark or when you need to use a specific version to support certain integrations or features. EMR and Databricks require a higher configuration and setup level than Glue. That involves choosing and configuring the instance types and sizes and selecting the software and framework versions. While this level of control can be beneficial in some cases, it also requires more time and expertise to get things set up and running smoothly.</p><p>EMR’s <strong>developer experience</strong> could be better, especially with Serverless. It is less intuitive and more laborious compared to Glue. That makes it harder to build and maintain data pipelines, especially for developers who are new to the platform.</p><p>Databricks has built-in support for <strong>Delta Lake</strong>, so you do not have to set up the dependencies or manage compatible versions. In addition, the Databricks runtime environment already includes <strong>many libraries </strong>commonly used, like Boto3, NumPy, and seaborn. That makes setting up and starting data processing easier, as you don’t need to install and manage these dependencies separately. It provides a <strong>collaborative notebook environment</strong> for Data Scientists, Data Engineers, and Analysts to work together on data projects.<strong> </strong>When we first tried it out, we were impressed by how easily the collaboration worked. Also, the fact that you have a REPL-like environment for SQL, Python, and Scala Code makes <strong>experimenting with data a breeze</strong>. Databricks has a very user-friendly interface and comprehensive documentation, making it easy for developers to get started.</p><h4><strong>Features</strong></h4><p>EMR supports a broader range of processing engines and big data frameworks, including not only Spark but also Hadoop, Flink, Trino, and others. That gives you more flexibility and choice when extending your data architecture.</p><p>One disadvantage of using EMR is that it requires an <strong>additional orchestration service</strong> to manage the data processing jobs and workloads like AWS Step Functions, AWS MWAA, or another tool. That adds a layer of complexity to your architecture and requires additional setup and maintenance. By contrast, Glue provides a natively integrated orchestration service, which makes it easier to set up and manage your ETL jobs.</p><p>As Glue, Databricks has <strong>built-in orchestration capabilities</strong>, allowing users to schedule and automate data pipelines. However, Databricks also allows for the orchestration of SQL queries, dbt, or even ML models.</p><p>The <strong>Unity Catalog</strong> in Databricks provides a centralized location for storing, accessing, and managing data assets. The catalog also includes <strong>lineage capabilities</strong>, allowing users to track the history of data transformations and the lineage of data assets on a column level. Additionally, Databricks inherently supports <strong>MLFlow</strong> for Data Scientists and offers <strong>serverless SQL processing</strong> for Business Analysts. That means we could switch out AWS Athena in the future, unleashing the power of all the Delta functionalities that Athena currently does not support.</p><h4><strong>Vendor Lock-In</strong></h4><p>Because EMR and Databricks have service-specific configurations, both come with a degree of vendor lock-in. Databricks offers more custom functionalities like Delta Live Tables, the Unity Catalog, or the proprietary processing engine Photon. Since there is no open-source version of those services, Databricks has more vendor lock-in than EMR.</p><h4><strong>Costs</strong></h4><p>EMR is generally cheaper than Glue, especially for large data processing tasks. There are two reasons for that. Firstly, compute hours simply cost less than using Glue. Secondly, EMR supports <strong>spot pricing</strong>, which allows you to bid on spare Amazon EC2 instances and potentially get a significant discount on your compute costs. Using spot instances is fine with Spark applications since it handles node failures in a fault-tolerant manner.</p><p>Databricks is also more cost-effective than AWS Glue, especially when leveraging AWS Graviton instances and enabling the vectorized query engine Photon that can substantially speed up large processing jobs. As EMR, Databricks supports spot instances, which reduce costs. Because it is a separate product from AWS, it has its own billing and pricing model, making it more annoying to manage and track expenses compared to using a single tool integrated with the rest of the AWS ecosystem.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/810/0*7nOh6a2BvizcFda0" /></figure><h4>EMR vs. Databricks</h4><p>In summary, Databricks and EMR are <strong>both mature and popular</strong> options for data processing and analysis in the cloud, making them<strong> </strong>valid replacements for AWS Glue. EMR has the advantage of supporting a broad range of processing engines and big data frameworks, as well as being <strong>generally cheaper</strong> than Databricks. Being part of the AWS ecosystem, EMR allows us to have unified billing and leverage our existing IAM integration to ensure the least privilege principle. However, it requires additional orchestration and data governance services and has a worse developer experience.</p><p>In contrast, Databricks has many features that <strong>increase developer productivity</strong> and <strong>enable Analysts and Data Scientists</strong> to work on the same platform and data while offering <strong>state-of-the-art security mechanisms</strong>. Nonetheless, it is more expensive while having the potential for vendor lock-in with proprietary technologies.</p><h4>Decision</h4><p>Comparing the two options, we concluded that <strong>Databricks</strong> would be the better fit for us. The few arguments speaking in favor of EMR, like unified billing and easy IAM integration, were not enough to convince us. What we liked especially about Databricks is the holistic approach they chose instead of the variety of thin products from AWS, like LakeFormation, MWAA, and Glue, that you would need to integrate first.</p><p>In the last part of this series, we will talk about the migration to Databricks and give our final resume on the journey.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/984/0*9fBXgeeCIXZcLsQE" /></figure><p><em>written by: </em><a href="https://www.linkedin.com/in/johannes-kotsch-28a874173"><em>Johannes Kotsch</em></a><em> &amp; </em><a href="https://www.linkedin.com/in/robert-kossendey-303b0019a"><em>Robert Kossendey</em></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=87e30c00b791" width="1" height="1" alt=""><hr><p><a href="https://medium.com/claimsforce/lakehouse-databricks-vs-aws-emr-87e30c00b791">Lakehouse — Databricks vs. AWS EMR</a> was originally published in <a href="https://medium.com/claimsforce">claimsforce</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Lakehouse — Running Delta Lake on AWS Glue]]></title>
            <link>https://medium.com/claimsforce/lakehouse-running-delta-lake-on-top-of-aws-glue-181133a916f3?source=rss-91105643a921------2</link>
            <guid isPermaLink="false">https://medium.com/p/181133a916f3</guid>
            <category><![CDATA[spark]]></category>
            <category><![CDATA[data-lakehouse]]></category>
            <category><![CDATA[aws-glue]]></category>
            <category><![CDATA[amazon-web-services]]></category>
            <category><![CDATA[delta-lake]]></category>
            <dc:creator><![CDATA[Robert Kossendey]]></dc:creator>
            <pubDate>Mon, 28 Nov 2022 09:39:18 GMT</pubDate>
            <atom:updated>2022-12-05T16:02:27.759Z</atom:updated>
            <content:encoded><![CDATA[<h3>Lakehouse — Running Delta Lake on AWS Glue</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rptWtWCZOr4Hs7rGFhcifg.png" /></figure><h4>Update 05.12.2022</h4><p>As <a href="https://medium.com/u/1ffeec546599">Noritaka Sekiyama</a> pointed out, Glue now supports Delta Lake natively with their latest update. That makes working with Delta Lake on AWS Glue easier but would not affect our decision if we had to decide again. The arguments against Glue, pricing, and lagging Spark versions, still stand.</p><h3>About this blog series</h3><p>At claimsforce, our initial approach to big data was a Two-Tier-Architecture consisting of a Data Lake stage in Amazon S3 and a Data Warehouse stage in Amazon Redshift (outline<a href="https://aws.amazon.com/blogs/startups/how-claimsforce-built-a-future-proof-lake-house-with-aws/"> here</a>). Over time we realized that having two stages comes with disadvantages like engineering and maintenance effort, infrastructure costs, and data staleness. We aim to replace the combination of a Data Lake and Data Warehouse with a unified system — the Lakehouse. In this blog series, we will document our journey toward a Lakehouse setup.</p><h3>Recapitulation</h3><p>The <a href="https://medium.com/claimsforce/lakehouse-the-journey-unifying-data-lake-and-data-warehouse-bef7629c143a">first article</a> of this blog series outlined the challenges we are facing with the traditional Two-Tier-Architecture. We discovered how the Lakehouse approach helps us overcome those challenges and decided to implement it using Delta Lake. This article will focus on the practical challenges of integrating Delta Lake into our existing data infrastructure that uses AWS Glue.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/670/0*gL8YZN0APe-oJpsp" /></figure><h3>AWS Glue</h3><p>Spark is the de-facto industry standard for processing large amounts of data. The Cloud Service Provider of our choice, AWS, provides scalable and flexible environments for running Spark applications. The multiple options to choose from range from DIY (running it on <strong>EC2</strong> machines) over managed clusters (<strong>EMR</strong>) to a fully serverless solution (<strong>Glue</strong>).</p><p>Glue offers the <strong>easiest-to-use</strong> execution environment for Spark. Although we knew that ease of usage comes with a tradeoff, we still chose to go with it, enabling us to get started writing code without thinking of managing the underlying infrastructure. Additionally, Glue provides well-integrated features like metadata cataloging or job scheduling.</p><p>The following describes which individual Glue features claimsforce uses and how:</p><ul><li>First, there is the Glue Data Catalog, a <strong>serverless</strong> Hive-compatible metastore. It consists of databases which in turn contain tables. A table is the metadata definition representing data in a data store, in our case Parquet files in S3. AWS Athena allows us to use that schema to <strong>directly query those tables</strong>, which would have been otherwise unstructured / semi-structured files in object storage.</li><li>We have a landing zone in <strong>S3</strong> that contains data dumped by production systems. To account for changing data and schemas, we do not enforce a schema on write but use Glue Crawler instead. These go through our tables and infer a schema, <strong>enabling dynamic table schema updates</strong>.</li><li>Then we use Glue Jobs, fully managed, <strong>serverless</strong> Apache Spark jobs for transforming and cleaning our raw data. That saves us the hassle of setting up and maintaining a Spark cluster. In addition, Glue provides a Framework on top of the Spark Context, aptly named Glue Context, extending on the built-in functionalities of Spark.</li><li>The last piece of the puzzle is Glue Workflows. It allows you to set up <strong>triggers</strong> for jobs, like cron expressions or another job succeeding. We use Workflows for the orchestration of our more complex pipelines.</li></ul><h3>Delta Lake + Glue</h3><p>Running Delta Lake on Glue is easy though it comes with some caveats. To set up a Glue Job that can read and write from/to a Delta Table, you need to add the Delta Jar to the Python Library Path and the Dependent Jar Path in the Job configuration. That allows you to use the DeltaTable API. If you additionally want to use Delta Commands like <strong>MERGE</strong> in Spark SQL, you can set the following config as a job parameter: “<em>spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension</em>”</p><p>Here is an example of a simple script that demonstrates writing and reading:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f78ee578e7a35f3a243b492a9b62fd29/href">https://medium.com/media/f78ee578e7a35f3a243b492a9b62fd29/href</a></iframe><p>Unfortunately, this approach has some restrictions. The biggest drawback is that Glue comes with a <strong>preconfigured Spark Version</strong>. Glue 3.0, the latest version, runs Spark 3.1.1, meaning that the only supported Delta versions are 1.0.X. <strong>None</strong> of the features <a href="https://www.databricks.com/blog/2022/06/30/open-sourcing-all-of-delta-lake.html">released</a> since 1.1 are currently available (<a href="https://docs.delta.io/2.0.0/optimizations-oss.html#z-ordering-multi-dimensional-clustering">ZOrder</a>, <a href="https://docs.delta.io/2.0.0/delta-change-data-feed.html">Change Data Feed</a>, <a href="https://docs.delta.io/2.0.0/delta-batch.html#dynamic-overwrites">Dynamic Partition Overwrites</a>). Even worse, bugs and security concerns are <strong>not patchable</strong>.</p><p>Since Glue is serverless and assigns pooled resources to each job, different jobs will likely have dissimilar Spark drivers. Concurrent writes to the same Delta table from multiple Spark drivers can lead to data loss. To prevent this, you can use a shared DynamoDB table as the <a href="https://docs.delta.io/latest/delta-storage.html#quickstart-s3-multi-cluster">LogStore</a> for all Glue Jobs.</p><h3>Our conclusion — Moving away from AWS Glue</h3><p>Besides the prior mentioned caveats, Glue is expensive, clocking in at <strong>$0.44</strong> for a DPU-Hour (4 vCPU, 16 GB Mem). For comparison, a Databricks job with one Driver and one Executor (same specs) would cost us <strong>$0.37</strong> per hour.</p><p>Furthermore, Glue’s built-in Meta Data Catalog is not usable for <strong>data discoverability</strong> purposes. While catering to the needs of Data Engineers, it fails to provide insights for Data Analysts, like data lineage or the latest update timestamps. This, and the fact that the Delta Lake integration is subpar, led us to the conclusion that it is time to abandon AWS Glue for a more mature and extensive product.</p><p>In the next part of this series, we will discuss the alternatives to Glue and our decision-making process.</p><p><em>written by: </em><a href="https://www.linkedin.com/in/johannes-kotsch-28a874173"><em>Johannes Kotsch</em></a><em> &amp; </em><a href="https://www.linkedin.com/in/robert-kossendey-303b0019a"><em>Robert Kossendey</em></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=181133a916f3" width="1" height="1" alt=""><hr><p><a href="https://medium.com/claimsforce/lakehouse-running-delta-lake-on-top-of-aws-glue-181133a916f3">Lakehouse — Running Delta Lake on AWS Glue</a> was originally published in <a href="https://medium.com/claimsforce">claimsforce</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Lakehouse — The journey unifying Data Lake and Data Warehouse]]></title>
            <link>https://medium.com/claimsforce/lakehouse-the-journey-unifying-data-lake-and-data-warehouse-bef7629c143a?source=rss-91105643a921------2</link>
            <guid isPermaLink="false">https://medium.com/p/bef7629c143a</guid>
            <category><![CDATA[data-lake]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[data-lakehouse]]></category>
            <category><![CDATA[data-warehouse]]></category>
            <category><![CDATA[delta-lake]]></category>
            <dc:creator><![CDATA[Robert Kossendey]]></dc:creator>
            <pubDate>Thu, 08 Sep 2022 07:20:19 GMT</pubDate>
            <atom:updated>2022-09-12T06:52:31.235Z</atom:updated>
            <content:encoded><![CDATA[<h3>Lakehouse — The journey unifying Data Lake and Data Warehouse</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*wa-r4tl3JDFjV2eM" /></figure><h3>About this blog series</h3><p>At claimsforce our initial approach to big data was a two-tier architecture consisting of a Data Lake stage in Amazon S3 and a Data Warehouse stage in Amazon Redshift (outline <a href="https://aws.amazon.com/blogs/startups/how-claimsforce-built-a-future-proof-lake-house-with-aws/">here</a>). Over time we realized that having two stages comes with several disadvantages like engineering and maintenance effort, infrastructure costs and data staleness. We aim to replace the combination of a Data Lake and Data Warehouse with a unified system — the Lakehouse. In this blog series we will document our journey towards a Lakehouse setup.</p><h3>Introduction</h3><p>At claimsforce our mission is to create great claims experiences. We believe that data can be leveraged to support the human decision maker and thrive to create the perfect human-machine interface in the claims area. Because of this we continuously question and reiterate on our big data infrastructure, the basis for all models and algorithms that fulfill our mission.</p><p>In this article we want to take the bird’s eye view. Firstly giving an overview of our current two-tier architecture and then outlining the challenges we face. Then we introduce the new Lakehouse architecture and discuss its advantages and disadvantages. To conclude the concrete set of technologies we use for our implementation are presented. The rest of the series will be about those concrete technologies.</p><h3>Our Status Quo — A Two-Tier Analytical Data Architecture</h3><p>Our current Data Architecture comprises two stages — A <strong>Data Lake</strong> and a <strong>Data Warehouse</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/778/0*cJ4SDzBGKj3aWn6V" /><figcaption>(<em>Simplified Version</em>)</figcaption></figure><p>We implemented a Data Lake on top of<strong> Amazon S3</strong>. Our operational systems offload their data into this stage. Rudimentary transformations such as column renaming and date standardization are performed as well at that stage.</p><p>The Data Lake Architecture paradigm has multiple <strong>advantages</strong>:</p><ul><li>Built on S3, it leverages the advantages of the service like <strong>low storage cost, infinite and seamless scalability, high durability, and availability.</strong></li><li>We write from and to the Data Lake with ETL jobs, <strong>separating storage and compute.</strong> That allows us to scale both aspects individually, enabling us to better respond to load peaks and reduce costs.</li><li>Since File-Storage doesn’t require a schema, we can write any file to S3 and later determine with which schema we want to read it (<strong>Schema-on-Read</strong>)</li><li>A Data Lake supports <strong>all</strong> data formats (also unstructured data like photos and videos)</li></ul><p>However, the Data Lake stage also comes with some <strong>disadvantages:</strong></p><ul><li><strong>Low query performance</strong> when performing aggregations on large amounts of data</li><li>No ACID transactions</li><li>Append-Only style of writing data. It is only easily possible to add new records of data. To be able to update existing records, it would be necessary to delete data and re-run the ETL scripts on all data.</li></ul><p>To make up for the downsides of the Data Lake stage, we added a second stage to our data infrastructure — The <strong>Data Warehouse</strong>. We used Amazon’s Data Warehouse implementation, <strong>Redshift</strong>. That allowed us to <strong>update existing rows, perform ACID transactions </strong>and <strong>query large amounts of data efficiently.</strong></p><p>Combining those two tiers of Data Architecture solved some of our problems because they supplement each other’s weaknesses. Integrating those two systems however created a new set of <strong>problems</strong>.</p><p>Firstly, we face increased maintenance efforts because there are two different types of infrastructure. In addition, there is a need to <strong>pay for a running Redshift cluster </strong>which is, since compute and storage are not separated, way more expensive than storing the data in a distributed file system like S3.</p><p>Internally, Data Warehouses like Redshift use proprietary data formats optimized for analytical queries on large amounts of data. However, extracting this data from the outside requires querying it via JDBC/ODBC, which is <strong>highly inefficient</strong>. Directly accessing the data stored in a Data Warehouse is not possible.</p><p>Since our Machine Learning use cases require work on refined datasets which sit in the Data Warehouse, it, unfortunately, became a <strong>bottleneck</strong> and severely slowed down the development cycle of our Machine Learning models.</p><h3>Our desired state — The Lakehouse</h3><p>The challenges we encounter with our data infrastructure have already been described by Michael Armbrust et al. in their paper “<a href="http://cidrdb.org/cidr2021/papers/cidr2021_paper17.pdf"><em>Lakehouse: A New Generation of Open Platforms that Unify Data Warehousing and Advanced Analytics</em></a><em>.</em>” To solve those problems, they propose an architecture type that unifies Data Lakes and Data Warehouses — The Lakehouse.</p><p>Open-source libraries like <a href="https://delta-io.translate.goog/?_x_tr_sl=en&amp;_x_tr_tl=de&amp;_x_tr_hl=de&amp;_x_tr_pto=sc"><strong>Delta Lake</strong></a>, <a href="https://iceberg.apache.org/"><strong>Apache Iceberg</strong></a>, or <a href="https://hudi.apache.org/"><strong>Apache Hudi</strong></a> allow implementing a transactional metadata layer, enabling <strong>ACID</strong> transactions, <strong>upserting</strong>, and <strong>versioning</strong> while keeping the data in your existing Data Lake. That allows for having the advantages of a DWH / DBMS, with the data being in a low-cost object store, separating storage from the computation.</p><p>In addition, those frameworks use existing open source data formats like parquet, enabling one to do machine learning directly on top of the files, requiring no additional reverse ETL.</p><h3>Our decision — Delta Lake + AWS Athena</h3><p>After comparing Iceberg, Hudi, and Delta Lake, we decided to go with <strong>Delta Lake</strong> (we highly recommend reading the paper <a href="https://databricks.com/wp-content/uploads/2020/08/p975-armbrust.pdf"><em>Delta Lake: High-Performance ACID Table Storage over Cloud Object Stores</em></a> by Michael Armbrust et al.).</p><p>Those are the reasons:</p><ul><li>Spark integration is great</li><li>Youngest of the three frameworks but already mature</li><li>Fast growing community</li><li>Backed by Databricks</li></ul><p>We believe no one-size-fits-all framework exists, and everyone should decide what to use based on their use case. There are some great comparison articles, which we will link at the bottom of the article.</p><p>As our query engine on top of the Lakehouse, we chose to go with AWS Athena for now since we already use it for ad-hoc analytical queries and it is cheap at 5 USD per TB. There are some disadvantages with Athena as a choice:</p><ul><li>Athena does unfortunately not integrate with Delta Lake <strong>natively yet</strong>. Athena cannot read the delta log, only the latest state of the table. Hence it can only provide a read-only interface to the Delta Table. Delta functionalities like updating, time travel or vacuum <strong>are not supported</strong> within Athena.</li><li>Athena can not scale as self-managed Presto/Trino clusters can; you will eventually hit a performance limit and resources can be temporarily unavailable.</li></ul><p>That marks the end of part one of our journey, moving away from the traditional Data Warehouse and fully embracing the Lakehouse. In part two of this series, we will discuss the challenges of implementing Delta Lake with the (spoiler) former ETL service of our choice, AWS Glue.</p><p><strong>Now it is time for the implementation!</strong></p><p><em>written by: </em><a href="https://www.linkedin.com/in/johannes-kotsch-28a874173"><em>Johannes Kotsch</em></a><em> &amp; </em><a href="https://www.linkedin.com/in/robert-kossendey-303b0019a"><em>Robert Kossendey</em></a></p><p><em>Related articles:</em></p><ul><li><a href="https://garystafford.medium.com/data-lake-table-formats-interest-and-adoption-rate-40817b87be9e">Open Source Data Lake Table Formats: Evaluating Current Interest and Rate of Adoption</a></li><li><a href="https://www.dremio.com/subsurface/comparison-of-data-lake-table-formats-iceberg-hudi-and-delta-lake/">Comparison of Data Lake Table Formats (Apache Iceberg, Apache Hudi and Delta Lake) | Dremio</a></li><li><a href="https://databricks.com/de/session_na20/a-thorough-comparison-of-delta-lake-iceberg-and-hudi">A Thorough Comparison of Delta Lake, Iceberg and Hudi - Databricks</a></li><li><a href="https://www.onehouse.ai/blog/apache-hudi-vs-delta-lake-vs-apache-iceberg-lakehouse-feature-comparison?830fc854_page=3">Onehouse</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bef7629c143a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/claimsforce/lakehouse-the-journey-unifying-data-lake-and-data-warehouse-bef7629c143a">Lakehouse — The journey unifying Data Lake and Data Warehouse</a> was originally published in <a href="https://medium.com/claimsforce">claimsforce</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>