<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Gustavo Martin Morcuende on Medium]]></title>
        <description><![CDATA[Stories by Gustavo Martin Morcuende on Medium]]></description>
        <link>https://medium.com/@gu.martinm?source=rss-ce76a0c7780b------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*yfsOvxs2C4TWNicuM3W-Mw.png</url>
            <title>Stories by Gustavo Martin Morcuende on Medium</title>
            <link>https://medium.com/@gu.martinm?source=rss-ce76a0c7780b------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 10:28:43 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@gu.martinm/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Lakehouse and Warehouse: sharing data between environments]]></title>
            <link>https://medium.com/adevinta-tech-blog/lakehouse-and-warehouse-sharing-data-between-environments-c2a76c478bdc?source=rss-ce76a0c7780b------2</link>
            <guid isPermaLink="false">https://medium.com/p/c2a76c478bdc</guid>
            <category><![CDATA[data-storage]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[lake-house]]></category>
            <category><![CDATA[data-platforms]]></category>
            <category><![CDATA[data-warehouse]]></category>
            <dc:creator><![CDATA[Gustavo Martin Morcuende]]></dc:creator>
            <pubDate>Tue, 10 Oct 2023 08:01:39 GMT</pubDate>
            <atom:updated>2023-10-10T08:01:39.592Z</atom:updated>
            <content:encoded><![CDATA[<h4><em>Simplifying access to data in production environments</em></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*c-usOG9bLzo2Y2KNd6b0hA.png" /></figure><h3>Introduction</h3><p>In <a href="https://adevinta.com/careers/our-locations/spain">Adevinta Spain</a>, we want to get the most out of the data that we manage. To achieve this goal, and due to the huge amounts of data to be stored, we had to implement storage systems based on Lakehouses and Data Warehouses.</p><p>Once the storage systems were in place we faced another problem: Data Engineers, Data Analysts, Data Scientists and other users want to explore and experiment with data located in the production environments in an easy way. We, on the other hand, want to isolate the production environments from the users so that they cannot damage the production processes running there.</p><p>In this article we will explain the solution that we have implemented to resolve this problem: protecting the production environments while still allowing users to use the data located there.</p><h3>Components for processing and consuming data</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*uyCf-X6iMLsoFJGn" /></figure><p>In this diagram we show all the basic components for processing and consuming data. Other diagrams may have the same components with different names — but the functionality is the same.</p><p>Let’s focus on the storage layer where we can find the Lakehouse and Data Warehouse.</p><h4>Storage layer</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*RwdpfDOpapeArRM1" /></figure><p>In the diagram you can see three different elements:</p><ul><li>Lakehouse: at Adevinta Spain we use <a href="https://aws.amazon.com/s3/">AWS S3</a>.</li><li>Data Warehouse: following our AWS platform choice, we use <a href="https://aws.amazon.com/redshift/">AWS Redshift</a>.</li><li>Metastore: in our case we are using <a href="https://hive.apache.org/">Hive Metastore</a> (we could also have used <a href="https://docs.aws.amazon.com/glue/latest/dg/catalog-and-crawler.html">Glue</a>).</li></ul><h3>Current situation (environment isolation)</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*a8QrdvFMkJvLiPcH" /></figure><p>Our production environment must be isolated from other environments because we do not want to break production processes. This creates different, isolated environments containing the same processing and storage components. So notebooks in the sandbox environment can not break data located in the integration or production environments.</p><h3>The problem with data</h3><p>The problem with having different and isolated environments is that, in pre-production environments, the amount of data available will probably be much lower than that which is generated in production.</p><p>Data Analysts, Data Scientists, Data Engineers, Machine Learning Engineers, and anyone working with big data, will need to work with enough data to develop and test their pipelines. Ideally, the development environment should offer the same amount of data that they would find in the production environment.</p><p>So now, we face the following problem: users want to be able to work with huge amounts of data in an easy and fast way. However, we want them to work in isolated environments because we do not want them to break anything in the production environment.</p><h4>The solution</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*h55yDJww0UBh1I4u" /></figure><p>We need to provide a solution where applications like notebooks running in isolated environments, such as integration, can access production storage.</p><h3>Lakehouse, AWS S3</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*SG4hjVrZF7v-yAi9" /></figure><p>When using a notebook (for example a Databricks notebook) and a metastore, the first thing it will do is ask the metastore where the data is physically located. Once the metastore responds, the notebook will go to the path in AWS S3 where the data is stored using the permissions given by the IAM Role.</p><h4>Lakehouse, sharing data with Waggle Dance</h4><p>Waggle Dance is a request routing Hive metastore proxy that allows tables to be concurrently accessed across multiple Hive deployments.</p><p>In short, Waggle Dance provides a unified endpoint with which you can describe, query and join tables that may exist in multiple distinct Hive deployments. Such deployments may exist in disparate regions, accounts, or clouds (security and network permitting).</p><p>Find out more <a href="https://github.com/ExpediaGroup/waggle-dance">here</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Y0uDuKzcQDEwQNYT" /></figure><p>The process breaks down like this:</p><ul><li>A laptop located in the integration environment requests a table</li><li>The Waggle Dance, also located in the integration environment, checks its configuration and decides whether the relevant metastore is located in the production or integration environment</li><li>The table request is routed to the correct metastore according to the Waggle Dance determination</li></ul><p>For example, this configuration could be based on some prefix. In the below example, the <strong>pro_</strong> prefix. When using this prefix the data to be retrieved will be located in the production environment instead of the integration one.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*t8wGyzg1Car9LubcYGNHZg.png" /></figure><h3>Data Warehouse on AWS Redshift</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*WKYtX4yJ_ty_F86d" /></figure><p>The Adevinta Spain Data Warehouse is implemented on top of the newer <a href="https://aws.amazon.com/redshift/features/ra3/">AWS Redshift RA3</a>. What makes RA3 different from the old Redshift is that, in the new implementation, computation and storage are separated.</p><h4>Data Warehouse, sharing data</h4><p>AWS Redshift RA3 includes a feature called Data Sharing. Using Data Sharing we can access data located on other Redshift servers and even in different accounts or environments with read-only permissions.</p><p><em>Data Sharing provides instant, granular and high-performance access without copying data or data movement. You can query live data constantly across all consumers on different RA3 clusters in the same AWS account, in a different AWS account or in a different AWS Region. Queries accessing shared data use the compute resources of the consumer Amazon Redshift cluster and don’t impact the performance of the producer cluster.</em></p><p>You can read more about Amazon Redshift RA3 here:</p><ul><li><a href="https://aws.amazon.com/blogs/big-data/use-amazon-redshift-ra3-with-managed-storage-in-your-modern-data-architecture/">https://aws.amazon.com/blogs/big-data/use-amazon-redshift-ra3-with-managed-storage-in-your-modern-data-architecture/</a></li><li><a href="https://aws.amazon.com/blogs/big-data/announcing-amazon-redshift-data-sharing-preview/">https://aws.amazon.com/blogs/big-data/announcing-amazon-redshift-data-sharing-preview/</a></li></ul><h4>Data Sharing</h4><p>With Data Sharing, we can configure AWS Redshift in the integration environment for accessing storage of the AWS Redshift instance located in the production environment.</p><p>You can find more information about how this works <a href="https://aws.amazon.com/blogs/big-data/sharing-amazon-redshift-data-securely-across-amazon-redshift-clusters-for-workload-isolation/">here</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*i5RnNXQTx0QzqMvn" /></figure><h4>Data Sharing, implementation</h4><p>The following steps show how to enable Data Sharing between integration and production AWS Redshift servers:</p><ol><li>In AWS Redshift RA3, production environment, run the following statements:</li></ol><ul><li>CREATE DATASHARE example_sharing</li><li>GRANT USAGE ON DATASHARE example_sharing TO ACCOUNT ‘INTEGRATION’</li><li>ALTER DATASHARE example_sharing ADD SCHEMA schema</li><li>ALTER DATASHARE example_sharing SET INCLUDENEW = TRUE FOR SCHEMA schema</li></ul><p>2. In AWS Redshift RA3, integration environment, run the following statements:</p><ul><li>CREATE DATABASE example_pro FROM DATASHARE exampe_sharing OF ACCOUNT ‘PRODUCTION’</li><li>CREATE EXTERNAL SCHEMA IF NOT EXISTS pro_schema FROM REDSHIFT DATABASE ‘example_pro’ SCHEMA ‘schema’;</li><li>GRANT USAGE ON SCHEMA pro_schema TO schema;</li></ul><p>With the above configuration, when using the <strong>pro_</strong> prefix in the integration environment, we will be accessing data located in production. This access is read only, so we cannot modify production data in any way.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gGVQDQ-hQJ71Kz9pXhS8Yg.png" /></figure><h3>Conclusion</h3><p>Through this article we have covered how to resolve the following problems in a Lakehouse and Data Warehouse implemented in AWS S3 and Redshift:</p><ul><li>Users (data engineers, data analysts, data scientists, etc, etc) need to work in pre-production environments with the same volume of data as in production.</li><li>We want to have different and isolated environments: integration, production etc.</li><li>Users need to work with the data in the easiest possible way — without compromising production systems or data</li></ul><p>In <a href="https://adevinta.com/careers/our-locations/spain">Adevinta Spain</a>, we have implemented these and other solutions in our Lakehouse and Data Warehouse with the primary objective of enhancing the user experience and reducing the workload required for creating <a href="https://medium.com/adevinta-tech-blog/evolving-from-a-data-platform-to-a-data-products-platform-c2008b9dcd34">data as products</a> with success. However, we don’t intend to stop here; our goal is to provide our users with the best possible experience. To achieve this, we plan to introduce additional solutions in areas such as cost control, access management, and more.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c2a76c478bdc" width="1" height="1" alt=""><hr><p><a href="https://medium.com/adevinta-tech-blog/lakehouse-and-warehouse-sharing-data-between-environments-c2a76c478bdc">Lakehouse and Warehouse: sharing data between environments</a> was originally published in <a href="https://medium.com/adevinta-tech-blog">Adevinta Tech Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to build your own data platform. Episode 2: authorization layer. Data Warehouse implementation.]]></title>
            <link>https://medium.com/@gu.martinm/how-to-build-your-own-data-platform-episode-2-authorization-layer-data-warehouse-implementation-ab1cbca04dfd?source=rss-ce76a0c7780b------2</link>
            <guid isPermaLink="false">https://medium.com/p/ab1cbca04dfd</guid>
            <category><![CDATA[lake-formation]]></category>
            <category><![CDATA[redshift]]></category>
            <category><![CDATA[authorization]]></category>
            <category><![CDATA[data-platforms]]></category>
            <category><![CDATA[data-mesh]]></category>
            <dc:creator><![CDATA[Gustavo Martin Morcuende]]></dc:creator>
            <pubDate>Sun, 04 Jun 2023 22:47:22 GMT</pubDate>
            <atom:updated>2023-06-04T22:47:22.525Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction.</h3><p>This article is the second part of the episode about building an authorization layer for your data platform. You can find the whole list of articles following this link: <a href="https://medium.com/@gu.martinm/list/how-to-build-your-own-data-platform-9e6f85e4ce39">https://medium.com/@gu.martinm/list/how-to-build-your-own-data-platform-9e6f85e4ce39</a></p><p>In the previous article we talked about how to implement the authorization layer in the Data Lake, in this second part we will be talking about the same but in the Data Warehouse.</p><h3>Authorization layer.</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qZ3eV1kBgpngye10XpBqPA.png" /></figure><p>You can see in this diagram the Lakehouse with its metastore and the Data Warehouse. We already talked about the authorization layer for the Lakehouse in the previous article. Now it is the turn for the Data Warehouse.</p><p>Because we will be using <a href="https://en.wikipedia.org/wiki/Amazon_Web_Services">Amazon Web Services</a> with <a href="https://aws.amazon.com/redshift/">AWS Redshift</a>, we will be implementing this layer using <a href="https://aws.amazon.com/lake-formation/">Lake Formation</a>.</p><h3>Processing layer.</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*azSmIckYjiH0MFk-2XYPDQ.png" /></figure><p>Human users and processes will be the ones accessing the stored data through the authorization layer. Machines and processes like <a href="https://docs.aws.amazon.com/es_es/emr/latest/ReleaseGuide/emr-zeppelin.html">Zeppelin notebooks</a>, <a href="https://aws.amazon.com/athena/">AWS Athena</a> for SQL, clusters of <a href="https://aws.amazon.com/emr/">AWS EMR</a>, <a href="https://www.databricks.com/">Databricks</a>, etc, etc.</p><h3>The problem with the authorization.</h3><p>Data engineers, data analysts and data scientists work in different and sometimes isolated teams. They do not want their data to be deleted or changed by tools or people outside their teams.</p><p>Data owners are typically in charge of granting access to their data.</p><h4>Owner — consumer, relationship.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/766/1*GBi_9nvvGBIIK46VlYaz8w.png" /></figure><ol><li>A data consumer requests access to some data owned by a different team in a different domain. For example, a table in a database.</li><li>The data owner grants access by approving the access request.</li><li>Upon the approval of an access request, a new permission is added to the specific table.</li></ol><p>Our authorization layer must be able to provide the above capability if we want to implement a data mesh with success.</p><h3>Data Warehouse, AWS Redshift.</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lKHuRc6OXoXn0ITdCzRitg.png" /></figure><p>The Data Warehouse is implemented on the top of AWS Redshift. Not many years ago a new service was released by Amazon called AWS Redshift RA3. What makes RA3 different from the old Redshift is that, in the new implementation, computation and storage are separated. Before having RA3, if users needed more storage capabilities, more computation had also to be paid even if computation was not a problem. And in the opposite way, when users needed more computation capabilities, more storage had to be paid. So, Redshift costs were typically high.</p><p>We will be using AWS Redshift RA3. Here you can find some useful links that explain further what are AWS Redshift and AWS Redshift RA3:</p><ul><li><a href="https://docs.aws.amazon.com/redshift/latest/mgmt/welcome.html">https://docs.aws.amazon.com/redshift/latest/mgmt/welcome.html</a></li><li><a href="https://aws.amazon.com/blogs/big-data/use-amazon-redshift-ra3-with-managed-storage-in-your-modern-data-architecture/">https://aws.amazon.com/blogs/big-data/use-amazon-redshift-ra3-with-managed-storage-in-your-modern-data-architecture/</a></li></ul><h3>Data Warehouse, AWS Redshift RA3.</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jPyFmHMEGv-84To3PnxnHg.png" /></figure><p>Amazon Redshift data sharing allows you to securely and easily share data for read purposes across different Amazon Redshift clusters without the complexity and delays associated with data copies and data movement. Data can be shared at many levels, including schemas, tables, views, and user-defined functions, providing fine-grained access controls that can be tailored for different users and businesses that all need access to the data.</p><p>Lake Formation can be integrated with data sharing.</p><p>For further information visit the following links:</p><ul><li><a href="https://aws.amazon.com/blogs/big-data/announcing-amazon-redshift-data-sharing-preview/">https://aws.amazon.com/blogs/big-data/announcing-amazon-redshift-data-sharing-preview/</a></li><li><a href="https://aws.amazon.com/blogs/big-data/centrally-manage-access-and-permissions-for-amazon-redshift-data-sharing-with-aws-lake-formation/">https://aws.amazon.com/blogs/big-data/centrally-manage-access-and-permissions-for-amazon-redshift-data-sharing-with-aws-lake-formation/</a></li></ul><h3>Authorization, Federated Lake Formation.</h3><p>Using Lake Formation with AWS Redshift RA3 we can manage the permissions across different accounts from only one central account in a federated way. We are delegating permissions to other accounts but we keep the control of them.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ks37DEvIVmD-3lAEDDPfeg.png" /></figure><h3>Authorization, implementation.</h3><p>In order to implement federated authorization with AWS Redshift RA3 you can follow the next steps:</p><ol><li>AWS Redshift RA3, producer account:</li></ol><ul><li>CREATE DATASHARE producer_sharing</li><li>GRANT USAGE ON DATASHARE producer_sharing TO ACCOUNT ‘FEDERATED_GOVERNANCE’</li><li>ALTER DATASHARE producer_sharing ADD SCHEMA producer_schema</li></ul><p>2. AWS Redshift RA3, consumer account:</p><ul><li>CREATE DATASHARE consumer_sharing</li><li>GRANT USAGE ON DATASHARE consumer_sharing TO ACCOUNT ‘FEDERATED_GOVERNANCE’</li><li>ALTER DATASHARE consumer_sharing ADD SCHEMA consumer_schema</li></ul><p>3. AWS Redshift RA3, main federated account:</p><ul><li>Through Lake formation console, allow access from consumer account to producer_sharing. You can see a screenshot about this configuration down below.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MmY6SCCyVnosu7e-oqSmAw.png" /></figure><p>With the above configuration, the query from the consumer account will only see the column <strong>brand_id</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yjaYEObT5eNhK6BELB7VlA.png" /></figure><h3>Conclusion.</h3><p>In this article we have explained how you can implement an authorization layer using AWS AWS Redshift RA3 and AWS Lake Formation.</p><p>With this authorization layer we will be able to resolve the following problems:</p><ul><li>Producers and consumers from different domains must have the capability of working in an isolated way (if they wish so) if we want to implement a data mesh with success.</li><li>Producers must be able to decide how consumers can access their data. They are the data owners, and they decide how others use their data.</li><li>Fine grained permissions can be established. At column and even if we want, at row level. This will be of great interest if we want to be <a href="https://en.wikipedia.org/wiki/General_Data_Protection_Regulation">GDPR</a> compliant. More information about how to implement the GDPR in your own data platform will be explained in future articles.</li></ul><p>Stay tuned for the next article about how to implement your own Data Platform with success.</p><p><em>I hope this article was useful. If you enjoy messing around with Big Data, Microservices, reverse engineering or any other computer stuff and want to share your experiences with me, just follow </em><a href="https://twitter.com/gumartinm"><em>me</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ab1cbca04dfd" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to build your own data platform. Episode 2: authorization layer. Data Lake implementation.]]></title>
            <link>https://medium.com/@gu.martinm/how-to-build-your-own-data-platform-episode-2-authorization-layer-data-lake-implementation-25f91270c360?source=rss-ce76a0c7780b------2</link>
            <guid isPermaLink="false">https://medium.com/p/25f91270c360</guid>
            <category><![CDATA[data-mesh]]></category>
            <category><![CDATA[spark]]></category>
            <category><![CDATA[data-platforms]]></category>
            <category><![CDATA[lake-formation]]></category>
            <category><![CDATA[authorization]]></category>
            <dc:creator><![CDATA[Gustavo Martin Morcuende]]></dc:creator>
            <pubDate>Fri, 02 Jun 2023 21:07:26 GMT</pubDate>
            <atom:updated>2023-06-02T21:07:26.024Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction.</h3><p>This is the second episode in the series about how to build your own data platform. You can find the whole list of articles in the following link <a href="https://medium.com/@gu.martinm/list/how-to-build-your-own-data-platform-9e6f85e4ce39">https://medium.com/@gu.martinm/list/how-to-build-your-own-data-platform-9e6f85e4ce39</a></p><p>Remember, a data platform will be used by many teams and users. Also the data to be stored could be coming from many and different sources. Data owners will want to set permissions and boundaries about who can access the data that they are creating in the data platform.</p><p>In this episode I will explain how you can add these capabilities to your data platform. Also I will introduce the concept of <a href="https://en.wikipedia.org/wiki/Data_mesh">data mesh</a>, and how you can use the authorization layer for implementing the workflow between data consumers and data owners that you will need for creating a successful data mesh.</p><h3>Authorization layer.</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*V-4SDhQhuEIMDxiH3A2qUQ.png" /></figure><p>Our authorization layer will be on the top of the storage one. In this way, users and applications willing to use the stored data will need to do it through this layer in a safe way. No data will escape from the storage layer without authorization.</p><p>For implementing this layer you can use different solutions like <a href="https://www.databricks.com/product/unity-catalog">Unity Catalog</a> from Databricks, <a href="https://aws.amazon.com/lake-formation/">Lake Formation</a> from AWS, plain <a href="https://docs.aws.amazon.com/IAM/latest/UserGuide/introduction.html">IAM</a> roles also from AWS, <a href="https://ranger.apache.org/">Apache Ranger</a>, <a href="https://privacera.com/">Privacera</a> and many others.</p><p>For this article, and because we are working with <a href="https://en.wikipedia.org/wiki/Amazon_Web_Services">Amazon Web Services</a>, we will be implementing this layer using IAM roles and Lake Formation.</p><h3>Processing layer.</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*92ogXwhMgEtkHCsLXN5KXQ.png" /></figure><p>Human users and processes will be the ones accessing the stored data through the authorization layer. In the processing layer we will fine applications like <a href="https://docs.aws.amazon.com/es_es/emr/latest/ReleaseGuide/emr-zeppelin.html">Zeppelin notebooks</a>, <a href="https://aws.amazon.com/athena/">AWS Athena</a> for SQL, clusters of <a href="https://aws.amazon.com/emr/">AWS EMR</a>, <a href="https://www.databricks.com/">Databricks</a>, etc, etc.</p><h3>The problem with the authorization.</h3><p>Data engineers, data analysts and data scientists work in different and sometimes isolated teams. They do not want their data to be deleted or changed by tools or people outside their teams.</p><p>Also, for being <a href="https://en.wikipedia.org/wiki/General_Data_Protection_Regulation">GDPR</a> compliant, to access <a href="https://en.wikipedia.org/wiki/Personal_data">PII</a> data, big restrictions will be required even at column or row level.</p><p>Every stored data needs to have an owner, and in Data Mesh, data owners are typically in charge of granting access to their data.</p><h3>What is a Data mesh?</h3><p>Taken from <a href="https://www.datamesh-architecture.com/#what-is-data-mesh">https://www.datamesh-architecture.com/#what-is-data-mesh</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qf9_zgUHq9wMxu0YNh7BWw.png" /></figure><p>The term data mesh, coined in 2019 by<a href="https://martinfowler.com/articles/data-mesh-principles.html"> Zhamak Dehghani</a>, is based on four key principles:</p><ol><li>Domain ownership: Domain teams are responsible for their data, aligning with the boundaries of their team’s domain. <strong>An authorization layer will be required for implementing those boundaries for some team.</strong></li><li>Data as a product: Analytical data should be treated as a product, with consumers beyond the domain. <strong>An owner-consumer relationship will exist, where consumers require access to products owned by a different team.</strong></li><li>Self-serve data infrastructure platform: A data platform team provides domain-agnostic tools and systems to build, execute, and maintain data products.</li><li>Federated governance: Interoperability of data products is achieved through standardization promoted by the governance group.</li></ol><h3>Owner — consumer, relationship.</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/756/1*0sq5e-fqmrOWR7WDNDjTIw.png" /></figure><ol><li>A data consumer requests access to some data owned by a different team in a different domain. For example, a table in a database.</li><li>The data owner grants access by approving the access request.</li><li>Upon the approval of an access request, a new permission is added to the specific table.</li></ol><p>Our authorization layer must be able to provide the above capability if we want to implement a data mesh with success.</p><h3>Data Lake.</h3><p>In this section we will write a brief recap about what we explained in previous article: <a href="https://medium.com/@gu.martinm/how-to-build-your-own-data-platform-f273014701ff">https://medium.com/@gu.martinm/how-to-build-your-own-data-platform-f273014701ff</a></p><h4>AWS S3.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qjSQagndbYzziSsEUX7ZeA.png" /></figure><ul><li>Notebooks, Spark jobs, clusters, etc, etc, run in Amazon virtual servers called EC2.</li><li>These virtual servers require permissions for accessing AWS S3. These permissions are given by IAM Roles.</li><li>We will be working with Amazon Web Services. As we said before, because the amount of data to be stored is huge, we can not use HDD or SSD data storages, we need something cheaper. In this case we will be talking about AWS S3.</li><li>Also, in order to ease the use of the Data Lake, we can implement metastores on the top of it. For example, Hive Metastore or Glue Catalog. We are not going to explain deeply how a metastore works, that will be left for another future article.</li></ul><p>When using a notebook (for example a Databricks notebook) and having a metastore, the first thing that the notebook will do is to ask the metastore where the data is physically located. Once the metastore responds, the notebook will go to the path in AWS S3 where the data is stored using the permissions given by the IAM Role.</p><h4>Direct access or with a metastore.</h4><p>We have two options for working with the data. With or without using a metastore.</p><p>With the metastore, users can have access to the data in the Data Lake in an easier way because they can use SQL statements as they do in any other databases.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HPMie6g4SIdWKVrPrVp1ng.png" /></figure><h3>Authorization, direct access.</h3><p>Consumers run their notebooks or any other applications from their AWS accounts and consume data located in the producer’s account.</p><p>These notebooks and applications run in Amazon virtual servers called Amazon EC2 instances, and for accessing the data located in AWS S3 in the producer’s account, they use IAM Roles (the permissions for accessing the data)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Kpeuq_Tkh7rfa-SCiZR6Dg.png" /></figure><h4>S3 bucket policy</h4><p>For example, for being able to access to the S3 bucket called <strong>s3://producer</strong>, with the IAM Role with <a href="https://docs.aws.amazon.com/IAM/latest/UserGuide/reference-arns.html">ARN</a> <strong>arn:aws:iam::ACCOUNT_CONSUMER:role/IAM_ROLE_CONSUMER</strong>, we can use the following AWS S3 <a href="https://docs.aws.amazon.com/AmazonS3/latest/userguide/bucket-policies.html">bucket policy</a> in the <strong>s3://producer</strong> bucket:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*amiIfTwj1SEre7kaz125Jw.png" /></figure><h4>Direct access</h4><p>Here, we are showing an example, where from a Databricks notebook using the above IAM Role and running in the consumer account, we are able to access data located in the producer’s account.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UtKCZ3EQ8TdfFB3A1BDg5A.png" /></figure><h4>Can we do it better?</h4><p>With <a href="https://docs.aws.amazon.com/glue/latest/dg/catalog-and-crawler.html">Glue Catalog as metastore</a>, data in S3 can be accessed as if it was stored in a table with rows and columns.</p><p>If we use tables instead of the direct access, we can grant permissions even at column level.</p><p><a href="https://aws.amazon.com/es/lake-formation/">Lake Formation</a> provides its own permissions model that augments the IAM permissions model. This centrally defined permissions model enables fine-grained access to data stored in data lakes through a simple grant or revoke mechanism, much like a database. Lake Formation permissions are enforced using granular controls at the column, row, and cell-levels.</p><h3>Authorization, Lake Formation.</h3><p>For using Lake Formation we will need the following elements:</p><ul><li>An application running in some machine in an AWS account. For example, an AWS EC2 instance where a Spark notebook will be executed.</li><li>A shared resource between the producer and consumer’s account. In this case we are sharing the S3 bucket called <strong>producer</strong>.</li><li>An IAM Role with permissions for using the producer’s bucket.</li><li>Two AWS Glue Catalogues as metastores. The one in the consumer’s account will be in charge of forwarding the table resolution to the metastore in the producer’s account. Both metastores are also shared between the two accounts.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gM6ZDh3rUWWhjjPmv7yEQg.png" /></figure><p>The catalogue in the producer’s account contains all the required information for translating the virtual table to its physical S3 location.</p><p>In the below screenshots you can see the Lake Formation configuration for the Glue metastore located in the producer’s account.</p><p>First you can see the table and database where the producer’s table is located. You can also see that we are sharing the specific table with the consumer’s account.</p><pre>Database: schema<br>Table: producer</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*V5tqrOAOeHkY-GBVLd9qJA.png" /></figure><p>In the above table we can configure access permissions. For example, we can decide that we will be allowing only the use of <strong>SELECT</strong> statements from the consumer’s account and also the only column that will be shown is the one called <strong>brand_id</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*opi_phVTj5PiM0v6lZzTUQ.png" /></figure><p>Now, from the Spark notebook running in the consumer’s account we can run SQL statements against the table located in the producer’s account.</p><p>Because we only allowed access to the column called <strong>brand_id</strong>, the consumer will only see values for that column. Any other column will be hidden.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UcX3d1ZaSr9guDMp6MCidw.png" /></figure><h3>Conclusion.</h3><p>In this article we have explained how you can implement an authorization layer using AWS IAM Roles and AWS Lake Formation.</p><p>With this authorization layer we will be able to resolve the following problems:</p><ul><li>Producers and consumers from different domains must have the capability of working in an isolated way (if they wish so) if we want to implement a data mesh with success.</li><li>Producers must be able to decide how consumers can access their data. They are the data owners, and they decide how others use their data.</li><li>Fine grained permissions can be established. At column and even if we want, at row level. This will be of great interest if we want to be <a href="https://en.wikipedia.org/wiki/General_Data_Protection_Regulation">GDPR</a> compliant. More information about how to implement the GDPR in your own data platform will be explained in future articles.</li></ul><p>Stay tuned for the next article about how to implement your own Data Platform with success.</p><p><em>I hope this article was useful. If you enjoy messing around with Big Data, Microservices, reverse engineering or any other computer stuff and want to share your experiences with me, just follow </em><a href="https://twitter.com/gumartinm"><em>me</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=25f91270c360" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to build your own data platform.]]></title>
            <link>https://medium.com/@gu.martinm/how-to-build-your-own-data-platform-f4bea25fbb98?source=rss-ce76a0c7780b------2</link>
            <guid isPermaLink="false">https://medium.com/p/f4bea25fbb98</guid>
            <category><![CDATA[data-mesh]]></category>
            <category><![CDATA[data-lakehouse]]></category>
            <category><![CDATA[redshift]]></category>
            <category><![CDATA[data-lake]]></category>
            <category><![CDATA[data-platforms]]></category>
            <dc:creator><![CDATA[Gustavo Martin Morcuende]]></dc:creator>
            <pubDate>Tue, 06 Dec 2022 00:03:44 GMT</pubDate>
            <atom:updated>2022-12-06T00:03:44.162Z</atom:updated>
            <content:encoded><![CDATA[<h3><a href="https://medium.com/@gu.martinm/how-to-build-your-own-data-platform-f273014701ff?source=user_profile---------0----------------------------">How to build your own data platform. Episode 1: sharing data between environments. Data Warehouse implementation.</a></h3><h3>Introduction.</h3><p>This article is the second part of the first episode about how to build your own data platform. To catch up, follow this link: <a href="https://medium.com/@gu.martinm/how-to-build-your-own-data-platform-f273014701ff">https://medium.com/@gu.martinm/how-to-build-your-own-data-platform-f273014701ff</a></p><p>As a short recap, remember that for creating a data platform many parts are involved. For this first episode we are only focusing on the component that we called storage layer. In the storage layer we could find the Lakehouse or Data Lake and the Data Warehouse. In the previous article we talked about how to share data in the Data Lake, in this second part we will be talking about the same but in the Data Warehouse.</p><h3>Storage layer.</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*4t7eXleH8r9UupQA" /></figure><p>You can see in this diagram three different elements:</p><ul><li><strong>Data Lakehouse</strong>: we already talked about it in the previous article.</li><li><strong>Metastore:</strong> also we explained it in the last article. We will talk about it more deeply in the coming articles.</li><li><strong>Data Warehouse:</strong> many times you will need to implement <a href="https://en.wikipedia.org/wiki/Star_schema">star schemas</a> for creating <a href="https://en.wikipedia.org/wiki/Data_mart">data marts</a>. Here, users can find meaningful data for creating dashboards, machine learning products or any other thing that users require. In this case, the Data Warehouse will be implemented on <a href="https://aws.amazon.com/redshift/">AWS Redshift</a>.</li></ul><h3>Current situation (environment isolation)</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*asJtRrvePlw7iSjB" /></figure><p>Remember that if you want users to create data products as fast as possible, you will need to create at least one environment where these users can mess around with the stored data. In this isolated environment they will be able to break and change as many things as they want. Our production environment must be isolated from this and other environments because we do not want to break productive processes.</p><h3>The problem with data.</h3><p>We want users to be able to work with huge amounts of data in an easy and fast way, but we want them to do that in isolated environments from the productive one because we do not want them to break anything.</p><h3>Data Warehouse, AWS Redshift.</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*SRlQWhLKQSzkliYQ" /></figure><p>All the environments have the same components but isolated one of each other.</p><p>The Data Warehouse is implemented on the top of AWS Redshift. Not many years ago a new service was released by Amazon called <a href="https://aws.amazon.com/redshift/features/ra3/">AWS Redshift RA3</a>. What makes RA3 different from the old Redshift is that, in the new implementation, computation and storage are separated. Before having RA3, if users needed more storage capabilities, more computation had also to be paid even if computation was not a problem. And in the opposite way, when users needed more computation capabilities, more storage had to be paid. So, Redshift costs were typically high.</p><p>Since the release of AWS Redshift RA3, because storage and computation are separated, users can decide if they want to increase either their storage or computational capabilities and only pay for what they need.</p><p>We will be using AWS Redshift RA3. Here you can find some useful links that explain further what are AWS Redshift and AWS Redshift RA3:</p><ul><li><a href="https://docs.aws.amazon.com/redshift/latest/mgmt/welcome.html">https://docs.aws.amazon.com/redshift/latest/mgmt/welcome.html</a></li><li><a href="https://aws.amazon.com/blogs/big-data/use-amazon-redshift-ra3-with-managed-storage-in-your-modern-data-architecture/">https://aws.amazon.com/blogs/big-data/use-amazon-redshift-ra3-with-managed-storage-in-your-modern-data-architecture/</a></li></ul><h3>Data Warehouse, Redshift RA3.</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*eLxqAO92SAUZqSns" /></figure><p>With Redshift RA3 storage is located under the component called Redshift Managed Storage located in AWS S3. As you can see on the above diagram, compute nodes are separated from the storage.</p><p>You can find more information about RA3 in the following link: <a href="https://aws.amazon.com/blogs/big-data/use-amazon-redshift-ra3-with-managed-storage-in-your-modern-data-architecture/">https://aws.amazon.com/blogs/big-data/use-amazon-redshift-ra3-with-managed-storage-in-your-modern-data-architecture/</a></p><h3>Data Warehouse, integration and production environments.</h3><p>In the integration environment we work with data as you can see in the below pictures.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*AFAeyFHb2C2qI6OV" /></figure><p>In the production environment we have the exact same system but isolated from the integration environment. In production we find the exact same statements.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*tehx7n1DBgqKfWaM" /></figure><h3>Data Warehouse, sharing data.</h3><p>AWS Redshift RA3 includes something called data sharing. With data sharing we can access with read only permissions to Redshift data located in other Redshift servers and even in different accounts or environments.</p><p><em>Data sharing provides instant, granular, and high-performance access without copying data and data movement. You can query live data constantly across all consumers on different RA3 clusters in the same AWS account, in a different AWS account, or in a different AWS Region. Queries accessing shared data use the compute resources of the consumer Amazon Redshift cluster and don’t impact the performance of the producer cluster.</em></p><ul><li><a href="https://aws.amazon.com/blogs/big-data/use-amazon-redshift-ra3-with-managed-storage-in-your-modern-data-architecture/">https://aws.amazon.com/blogs/big-data/use-amazon-redshift-ra3-with-managed-storage-in-your-modern-data-architecture/</a></li><li><a href="https://aws.amazon.com/blogs/big-data/announcing-amazon-redshift-data-sharing-preview/">https://aws.amazon.com/blogs/big-data/announcing-amazon-redshift-data-sharing-preview/</a></li></ul><h4>Data Sharing</h4><p>With Data Sharing, we can configure the AWS Redshift in the integration environment for accessing the storage of the AWS Redshift located in the production environment.</p><p>You can find more information about it in the following link: <a href="https://aws.amazon.com/blogs/big-data/sharing-amazon-redshift-data-securely-across-amazon-redshift-clusters-for-workload-isolation/">https://aws.amazon.com/blogs/big-data/sharing-amazon-redshift-data-securely-across-amazon-redshift-clusters-for-workload-isolation/</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*FKakFNvVWp9Btvaz" /></figure><h4>Data Sharing, implementation</h4><p>In order to create a data sharing between the integration and production AWS Redshift servers, you can follow the next steps.</p><ol><li>AWS Redshift RA3, production environment, statements to run:</li></ol><ul><li>CREATE DATASHARE meetup_sharing</li><li>GRANT USAGE ON DATASHARE meetup_sharing TO ACCOUNT ‘INTEGRATION’</li><li>ALTER DATASHARE meetup_sharing ADD SCHEMA schema</li><li>ALTER DATASHARE meetup_sharing SET INCLUDENEW = TRUE FOR SCHEMA schema</li></ul><p>2. AWS Redshift RA3, integration environment, statements to run:</p><ul><li>CREATE DATABASE meetup_pro FROM DATASHARE meetup_sharing OF ACCOUNT ‘PRODUCTION’</li><li>CREATE EXTERNAL SCHEMA IF NOT EXISTS pro_schema FROM REDSHIFT DATABASE ‘meetup_pro’ SCHEMA ‘schema’;</li><li>GRANT USAGE ON SCHEMA pro_schema TO schema;</li></ul><p>With the above configuration, when using the <strong>pro_</strong> prefix in the integration environment, we will be accessing data located in the production one. This access is read only, so we can not modify that data in any way.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*nrl5vykEfSiE7zaj" /></figure><h3>Conclusion.</h3><p>Through this article we have covered how to resolve the following problems in a Data Warehouse implemented in AWS Redshift RA3:</p><ul><li>Users (data engineers, data analysts and data scientists) need to work in pre-production environments with the same amount of data as in production.</li><li>We want to have different and isolated environments: integration, production, etc.</li><li>Users need to work with the data in the easiest possible way.</li></ul><p>Stay tuned for the next article about how to implement your own Data Platform with success.</p><p><em>I hope this article was useful. If you enjoy messing around with Big Data, Microservices, reverse engineering or any other computer stuff and want to share your experiences with me, just follow </em><a href="https://twitter.com/gumartinm"><em>me</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f4bea25fbb98" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to build your own data platform.]]></title>
            <link>https://medium.com/@gu.martinm/how-to-build-your-own-data-platform-f273014701ff?source=rss-ce76a0c7780b------2</link>
            <guid isPermaLink="false">https://medium.com/p/f273014701ff</guid>
            <category><![CDATA[data-management-platform]]></category>
            <category><![CDATA[data-lakehouse]]></category>
            <category><![CDATA[data-lake]]></category>
            <category><![CDATA[data-mesh]]></category>
            <category><![CDATA[data-platforms]]></category>
            <dc:creator><![CDATA[Gustavo Martin Morcuende]]></dc:creator>
            <pubDate>Tue, 29 Nov 2022 01:08:34 GMT</pubDate>
            <atom:updated>2022-12-05T23:48:06.647Z</atom:updated>
            <cc:license>http://creativecommons.org/publicdomain/zero/1.0/</cc:license>
            <content:encoded><![CDATA[<h3>How to build your own data platform. Episode 1: sharing data between environments. Data Lake implementation.</h3><h3>Introduction.</h3><p>Data is the new oil. Companies want to make the most of the data they produce. For achieving this goal, there is a need for systems capable of consuming, processing, analysing, and presenting massive volumes of data. These systems need to be easy to use, but they also need to be reliable, able to detect problems and store data correctly. These and other issues are intended to be resolved by Data Platforms.</p><p>It is not an easy task to build a Data Platform. Multiple skill sets are needed, from infrastructure and programming to data management.</p><p>This article is the first, of what I hope will be a longer series of articles where we’ll try to unravel the secrets of how to build a Data Platform that allows you to generate value-added products for your users.</p><h3>What is a data platform?</h3><p>We can discover definitions of what is a data platform just using our preferred web search engine. For example, I found the following definitions:</p><ul><li><strong>A data platform enables the acquisition, storage, preparation, delivery, and governance of your data, and adds a security layer for users and applications: </strong><a href="https://www.mongodb.com/what-is-a-data-platform">https://www.mongodb.com/what-is-a-data-platform</a></li><li><strong>A data platform is a complete solution for ingesting, processing, analyzing and presenting the data generated by the systems, processes and infrastructures of the modern digital organization: </strong><a href="https://www.splunk.com/en_us/data-insider/what-is-a-data-platform.html">https://www.splunk.com/en_us/data-insider/what-is-a-data-platform.html</a></li></ul><p>So, a data platform is a place where we can store data from multiple sources. Also a data platform provides users with the required tools for searching, working and transforming that data, with the goal of creating some kinds of products. These products could be dashboards with useful insights, machine learning products, etc, etc.</p><h3>What is a data platform? Very simplified diagram.</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*uIzCFm-CbTSn1kQe" /></figure><p>In this diagram we can find all the basic components that create a data platform (we are not trying to describe a <a href="https://en.wikipedia.org/wiki/Data_mesh">Data Mesh</a> or a <a href="https://en.wikipedia.org/wiki/Data_management_platform">Data Management Platform</a>, those things will be left out for other future articles) You can find the same components with other names but same functionality in other diagrams describing other data platforms. In this diagram we can find these components:</p><ul><li><strong>Data Sources:</strong> databases, REST APIs, event buses, analytics tools, etc, etc.</li><li><strong>Consumption: </strong>tools for consuming the data sources.</li><li><strong>Storage:</strong> the place where the consumed data will be located.</li><li><strong>Security layer:</strong> component in charge of providing authentication, authorization and auditory.</li><li><strong>Processing:</strong> programs or tools that will enable us to work with the stored data.</li><li><strong>Data Catalog:</strong> because the amount of stored data will be huge, we need a tool that will make easy for users to find the data that they need.</li><li><strong>Tableau, Qlik, Kubeflow, MLflow, etc, etc:</strong> data will be used for some goal. Typically this goal could be to create a dashboard with meaningful diagrams, create models for machine learning and many other things.</li></ul><p>This first article will be focusing on the storage layer, so from now on, we will talk only about that component.</p><h3>Storage layer.</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Rcf85topzTCiBy9Y" /></figure><p>Of course, the storage layer is the place where the data is stored. Because the amount of data to be stored is huge, we can not use HDD or SSD data storages, we need something cheaper. In this case we will be talking about <a href="https://aws.amazon.com/s3/">AWS S3</a> because we are working with <a href="https://en.wikipedia.org/wiki/Amazon_Web_Services">Amazon Web Services</a>. For <a href="https://en.wikipedia.org/wiki/Microsoft_Azure">Azure</a>, you could use <a href="https://learn.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-introduction">Azure Data Lake Storage Gen2</a>. If you are working with <a href="https://en.wikipedia.org/wiki/Google_Cloud_Platform">Google Cloud</a>, you could use <a href="https://cloud.google.com/storage">Google Cloud Storage</a>. It does not matter what storage you use as long as it is cheap and can store a huge amount of data.</p><p>You can see in this diagram three different elements:</p><ul><li><strong>Data Lakehouse:</strong> it is the evolution of the traditional Data Lake. Data Lakehouse implements all the capabilities of a Data Lake plus ACID transactions. You can find more information about Lakehouses in this <a href="https://www.databricks.com/glossary/data-lakehouse">link</a>.</li></ul><p>Usually in a Data Lakehouse or a Data Lake you can find different zones for storing data. The number of zones that you can find depends on how you want to classify your data. How to create and classify the data in your Data Lake or Lakehouse is a complicated matter that will be treated in a future article. The Data Lake is the first place where the consumed data is stored. Sometimes it is just meaningless raw data.</p><ul><li><strong>Data Warehouse:</strong> many times you will need to implement <a href="https://en.wikipedia.org/wiki/Star_schema">star schemas</a> for creating <a href="https://en.wikipedia.org/wiki/Data_mart">data marts</a> in order to make easy for users the use of the stored data. Here, users can find meaningful data for creating dashboards, machine learning products or any other thing that users require.</li><li><strong>Metastore:</strong> data is stored in the blob storage, if you want to use this data as if it was stored in a traditional database we need an element for translating schemas and table names to folders and files in the blob storage. This translation is made by the metastore.</li></ul><p>This article does not try to deeply explain how the above three elements work. Those explanations will be left out for other future articles.</p><h3>Current situation (environment isolation)</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*zHWL3Wx2W6dRQT5u" /></figure><p>If you want users to create data products as fast as possible, you will need to create at least one environment where these users can mess around with the stored data. In this isolated environment they will be able to break and change as many things as they want. Our production environment must be isolated from this and other environments because we do not want to break productive processes. Different and isolated environments will exist. These environments contain the same processing and storage layers but these layers are isolated in their own environments. So notebooks in the sandbox environment can not break data stored in the storage layer from the production environment.</p><h3>The problem with data.</h3><p>Data engineers, data analysts, data scientists and data people in general who work with big data require huge amounts of data in order to implement, develop and test their applications. These applications could be ETLs, ELTs, notebooks, dashboards, etc, etc.</p><p>In a healthy system, these users should be able to work in a safe environment where they can be sure <strong>that they do not break anything that already works in production when trying to implement new solutions</strong>. We need to create isolated environments. These environments could be sandbox, integration, production environment, etc, etc.</p><p>The problem with having different and isolated environments is that, in no productive environments, the amount of data will probably be much lower than the one that will be generated in production.</p><p>So now, we face the following problem: <strong>we want users to be able to work with huge amounts of data in an easy and fast way, but we want them to do that in isolated environments from the productive one because we do not want them to break anything</strong>.</p><h3>The solution 1.</h3><p>Remember that we have data sources, and those data sources must be connected to our different and isolated environments. We could ask those data sources to send us the same amount of data as they are sending in the productive environment.</p><p>The problem with this solution is that, in many cases, those data sources have their own no preproductive environments and it is impossible for them to generate the same amount of data in the rest of the environments as in the production environment. Also, they will not be willing to connect our own no preproductive environments to their productive ones because we could break their environments.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*tOOaqUDXKhMxRXIY" /></figure><p><strong>This solution in many cases will not work.</strong></p><h3>The solution 2.</h3><p>Another solution could be as simple as implementing a job for copying data from the storage located in the productive environment to the no productive one. For example, a Jenkins job.</p><p>The problem with this solution is that copying huge amounts of data is not fast and also, the job can break easily for multiple reasons (not having the right permissions, the right amount of memory for moving all the required data, etc, etc)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*GrGRfX3uLHoP7xUc" /></figure><p><strong>This solution does not ease the development of new applications because the copying process is slow, sometimes will not work, and data is not immediately available.</strong></p><h3>The solution 3.</h3><p>What our users need is to have access to data generated in the production environment from the tools running in the no productive environments. We need to provide a solution where applications like notebooks running in for example the integration environment can access the storage located in the productive one.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*W-da05ayv9WuwitT" /></figure><p><strong>This solution will work in all cases. This is the solution that we are going to explain in this article focusing on the component related to the Data Lake. In a next article we will explain the same solution implemented in a Data Warehouse.</strong></p><h3>Data Lake, AWS S3.</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*-qlnXqJhrjM7lrs2" /></figure><ul><li>Notebooks, Spark jobs, clusters, etc, etc, run in Amazon virtual servers called <a href="https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/concepts.html">EC2</a>.</li><li>These virtual servers require permissions for accessing AWS S3. These permissions are given by <a href="https://docs.aws.amazon.com/IAM/latest/UserGuide/introduction.html">IAM Roles</a>.</li><li>We will be working with Amazon Web Services. As we said before, because the amount of data to be stored is huge, we can not use HDD or SSD data storages, we need something cheaper. In this case we will be talking about <a href="https://aws.amazon.com/s3/">AWS S3</a>.</li><li>Also, in order to make easy the use of the Data Lake, we can implement <a href="https://www.quora.com/What-is-hive-meta-store/answer/Shruti-Pawar-9">metastores</a> on the top of it. For example, Hive Metastore or Glue Catalog. We are not going to explain deeply how a metastore works, that will be left for another future article.</li></ul><p>When using a notebook (for example a Databricks notebook) and having a metastore, the first thing that the notebook will do is to ask the metastore where the data is physically located. Once the metastore responds, the notebook will go to the path in AWS S3 where the data is stored using the permissions given by the IAM Role.</p><h3>Data Lake, integration and production environments.</h3><p>In the integration environment we have two options for working with the data. With or without using a metastore.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*1yY2m_hh4l_tfcVC" /></figure><p>In the production environment we have the exact same system but isolated from the integration environment. In production we find the exact same two options.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*MNkiD0DUs4lq12SO" /></figure><p>As you all can see, the metastore allows us to use the data located in the Data Lake as it was a normal database. Also, we can see that the metastore does not store data but the metadata that allows us to find the real stored data in AWS S3. With the metastore, users can have access to the data in the Data Lake in an easier way because they can use SQL statements as they do in any other database.</p><h3>Data Lake, sharing data.</h3><p>When users run their notebooks or any other application from the integration environment they need to have access to the production data located in the storage zone of the production environment.</p><p>Remember that those notebooks and applications run in Amazon virtual servers called Amazon EC2 instances, and for accessing the data located in AWS S3 they use IAM Roles (the permissions for accessing the data) We can modify the IAM Role in the (for example) integration environment in order to allow EC2 instances to access data located in the productive storage zone.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*seeLOi36HKVmXjis" /></figure><h4>IAM Role configuration.</h4><p>For example, for being able to access to S3 integration and production folders we can configure the IAM Role in the following way:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*H2oSAFuKq-U6wokC" /></figure><p>Any application running on a machine with this IAM Role can read data from production and integration and can only modify the data located in the integration environment. So the productive data is not modified in any way.</p><h4>Applying the solution.</h4><p>Once we have applied the above configuration in the IAM Role, users have direct access to the data located in the productive environment, for example from the integration environment.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*zNCxzQz4JDcwp68n" /></figure><h4>Can we do it better?</h4><p>With this configuration users, from for example their notebooks, can access the productive data and work with it without being able to modify it. But, we know, by means of a metastore, users can access the data even in an easier way. So the question is: can we use metastores with this solution?</p><p>We will see how to do it in the next section of this article.</p><h3>Data Lake, sharing data. Waggle Dance.</h3><p>Waggle Dance is a request routing Hive metastore proxy that allows tables to be concurrently accessed across multiple Hive deployments.</p><p>In short, Waggle Dance provides a unified endpoint with which you can describe, query, and join tables that may exist in multiple distinct Hive deployments. Such deployments may exist in disparate regions, accounts, or clouds (security and network permitting).</p><p>For further information follow this link: <a href="https://github.com/ExpediaGroup/waggle-dance">https://github.com/ExpediaGroup/waggle-dance</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*l_QEBYamQhs7OYiJ" /></figure><p>Now, when asking for some table from the integration environment, and based on some configuration, the Waggle Dance living in the integration environment decides if the metastore to be asked resides either in the production or integration environment.</p><p>For example, this configuration could be based on some prefix. In the below example, the <strong>pro_</strong> prefix. When using this prefix the data to be retrieved will be located in the production environment instead of the integration one.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*vqRWZQ0a4IMLALpI" /></figure><h3>Conclusion.</h3><p>Through this article we have covered how to resolve the following problems in a Data Lake implemented in AWS S3:</p><ul><li>Users (data engineers, data analysts and data scientists) need to work in pre-production environments with the same amount of data as in production.</li><li>We want to have different and isolated environments: integration, production, etc.</li><li>Users need to work with the data in the easiest possible way.</li></ul><p>Stay tuned for the next article about how to share data with AWS Redshift and many others that will follow about how to implement your own Data Platform with success.</p><p><em>I hope this article was useful. If you enjoy messing around with Big Data, Microservices, reverse engineering or any other computer stuff and want to share your experiences with me, just follow </em><a href="https://twitter.com/gumartinm"><em>me</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f273014701ff" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[PySpark: unit, integration and end-to-end tests.]]></title>
            <link>https://medium.com/@gu.martinm/pyspark-unit-integration-and-end-to-end-tests-c2ba71467d85?source=rss-ce76a0c7780b------2</link>
            <guid isPermaLink="false">https://medium.com/p/c2ba71467d85</guid>
            <category><![CDATA[end-to-end-testing]]></category>
            <category><![CDATA[pyspark]]></category>
            <category><![CDATA[unit-testing]]></category>
            <category><![CDATA[pytest]]></category>
            <category><![CDATA[spark]]></category>
            <dc:creator><![CDATA[Gustavo Martin Morcuende]]></dc:creator>
            <pubDate>Sun, 30 Aug 2020 19:16:26 GMT</pubDate>
            <atom:updated>2020-08-30T19:58:40.969Z</atom:updated>
            <cc:license>http://creativecommons.org/publicdomain/zero/1.0/</cc:license>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*oAPXBui9xBtx0Q00ArTFMw.png" /></figure><h3>Introduction</h3><p>Many years ago software developers understood the benefits of testing their code. Testing in the big data world is not so extended. Through this article I intend to show one way of creating and running PySpark tests.</p><p>There are many articles out there where it is explained how to write tests and integrate them in the CI pipelines. When working with Spark I did not manage to find any good documentation or patterns that could help me to create and automate tests in the same way as I used to do with other frameworks.</p><p>This article explains the way I run PySpark tests. Hopefully, it will be useful for future big data developers searching ways to improve the quality of their code and at the same time their CI pipelines.</p><h3>Unit, integration and end-to-end tests.</h3><p>These are the kinds of tests that we can need when working with Spark. I am not mentioning others like smoke tests, acceptance tests, etc, etc because I think they are outside the scope of this article.</p><ul><li><strong>Unit Tests:</strong> at this level we will be dealing with code that does not require a Spark Session in order to work. Also, this kind of code does not talk with the outside world.</li><li><strong>Integration Tests:</strong> at some point we will need to use a Spark Session. At this level we will be testing Spark transformations and in many cases we will have to deal with external systems such as databases, Kafka clusters, etc, etc.</li><li><strong>End-to-end Tests: </strong>our application probably will be composed of several Spark transformations working together in order to implement some feature required by some user. Here, we will be testing the whole application.</li></ul><p>You must be wondering why I am not drawing the typical test pyramid. Well, the answer is simple: most of the time, when working with data we will be dealing with integration tests instead of unit tests so the test pyramid does not make too much sense in our case.</p><h3>PySpark project layout</h3><p>This is a strongly opinionated layout so do not take it as if it was the only and best solution. I think this layout should work under any use case but if it does not work for you, at least I hope, it will bring some inspiration or ideas to your testing implementation.</p><pre>├── src<br>│   └── awesome<br>│       ├── app<br>│       │   └── awesome_app.py<br>│       ├── job<br>│       │   └── awesome_job.py<br>│       └── service<br>│           └── awesome_service.py<br>└── tests<br>    ├── endtoend<br>    │   ├── fixtures<br>    │   └── test_awesome_app.py<br>    ├── integration<br>    │   ├── fixtures<br>    │   └── test_awesome_job.py<br>    ├── shared_spark_session_helper.py<br>    └── unit<br>        └── test_awesome_service.py</pre><p>It is a <a href="https://docs.pytest.org/en/stable/goodpractices.html">src layout</a> using <a href="https://docs.pytest.org/en/stable/"><strong>pytest</strong></a> with three different packages: app, job and service for our PySpark application. For pytest we will be using three different folders: endtoend, integration and unit.</p><h3>Application layout</h3><h4><strong>app package</strong></h4><p>Under this folder we will find the modules in charge of running our PySpark applications. Typically we will have only one PySpark application. For example, in the above layout,<em> awesome_app.py</em> will contain the <em>__main__</em> required for running the application.</p><h4>job package</h4><p>A Spark application should implement some kind of transformations. Modules under this package run Spark jobs that require a Spark Session. For example, <em>awesome_job.py</em> could contain Spark code implementing one or several transformations.</p><h4>service package</h4><p>Sometimes business logic does not require a Spark Session in order to work. In such cases, we can implement the logic in a different module.</p><h3>Pytest layout</h3><h4>unit folder</h4><p>Folder where our unit tests will reside. Understanding unit tests as those that neither require a Spark Session in order to work nor talk with the outside world (file systems, databases, etc, etc)</p><h4>integration folder</h4><p>Contains tests that require a Spark Session or deal with the outside world. There is also a fixture folder where we can store the required data sets for running our tests. Under this path we can find tests that check Spark transformations.</p><h4>endtoend folder</h4><p>This folder includes tests that run a whole PySpark application and check that results are correct. Spark applications can be composed of multiple transformations, tests under this path check the application as a whole. As in the integration folder, there is a fixture folder where we can include data sets for testing our applications.</p><h3>Shared Spark Session</h3><p>One of the biggest problems to be solved when running Spark tests is the isolation of these tests. Running a test should not affect the results of another. In order to achieve this goal we are going to need a Spark Session for each set of tests, in this way, the results of these tests will not affect others that will also require a Spark Session.</p><p>So, we need to implement a system that will enable us to run, clear and stop a Spark Session whenever we need it (before and after a set of related Spark tests)</p><p>As stated before, we will be using pytest. Pytest works with fixtures but also allows us to use a <a href="https://docs.pytest.org/en/stable/xunit_setup.html#classic-xunit-style-setup">classic xunit style setup</a>. We will be using this xunit style for our Shared Spark Session. The details of the implementation are explained down below:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5e1c6672a5ca9aa363b7c71030bdda4c/href">https://medium.com/media/5e1c6672a5ca9aa363b7c71030bdda4c/href</a></iframe><ul><li><strong>setup_class method:</strong> we want to share the same Spark Session across a set of tests. The boundaries of this set will be represented by a test class, what means, in one test class we can share the same Spark Session through all the tests located in that test class.</li><li><strong>spark_conf class method:</strong> different set of tests may also need Spark Sessions with different configurations. The <em>spark_conf</em> method enables us to load a Spark Session with the required configuration for each set of tests.</li><li><strong>Embedded Hive:</strong> <em>spark-warehouse</em> and <em>metastore_db</em> are folders used by Spark when enabling the <a href="https://jaceklaskowski.gitbooks.io/mastering-spark-sql/content/spark-sql-hive-metastore.html">Hive support</a>. Different Spark Sessions in the same process can not use the same folders. Because of that, we need to create random folders in every Spark Session.</li><li><strong>setup_method</strong>: creates a temporary path which is useful when our Spark tests end up writing results in some location.</li><li><strong>teardown_method</strong>: clears and resets the Spark Session at the end of every test. Also, it removes the temporary path.</li><li><strong>teardown_class method</strong>: stops the current Spark Session after the set of tests are run. In this way we will be able to run a new Spark Session if it is needed (if there is another set of tests requiring the use of Spark)</li></ul><h4>How it works</h4><p>The basic idea behind <em>SharedSparkSessionHelper </em>lies in the fact that there is one Spark Session per Java process and it is stored in an <a href="https://docs.oracle.com/javase/8/docs/api/java/lang/InheritableThreadLocal.html"><em>InheritableThreadLocal</em></a>. When calling <a href="https://spark.apache.org/docs/latest/api/java/org/apache/spark/sql/SparkSession.Builder.html"><em>getOrCreate</em></a> method from <em>SparkSession.Builder</em> we end up either creating a new Spark Session (and storing it in the <em>InheritableThreadLocal</em>) or using an existing one.</p><p>So, for example, when running an end-to-end test, because <em>SharedSparkSessionHelper </em>is loaded before anything else (by means of the <em>setup_class</em> method), the application under test will be using the Spark Session launched by <em>SharedSparkSessionHelper</em>.</p><p>Once the test class is finished, the <em>teardown_class</em> method stops the Spark Session and removes it from the <em>InheritableThreadLocal </em>leaving our test environment ready for a new Spark Session. In this way, tests using Spark can run in an isolated way.</p><h3>Awesome project</h3><p>This article would be nothing without a real example. Just following this <a href="https://github.com/gumartinm/pyspark-shared-spark-session-helper">link</a> you will find a project with <a href="https://tox.readthedocs.io/en/latest/">tox</a>, <a href="https://github.com/pypa/pipenv">pipenv</a>, <a href="https://docs.pytest.org/en/stable/">pytest</a>, <a href="https://www.pylint.org/">pylint</a> and <a href="https://github.com/PyCQA/pycodestyle">pycodestyle</a> where I use the <em>SharedSparkSessionHelper</em> class.</p><p>The Awesome project uses object-oriented programming style but the same could be applied when using procedural style, the only requirement is to write test classes which can be used with the <em>SharedSparkSessionHelper</em> class. Many times you will end up using either object-oriented or procedural style or both at the same time in your python projects. Feel free to use one or another depending on what you need.</p><p>Of course this application can be run in any of the available clusters that currently exist such as <a href="https://kubernetes.io/">Kubernetes</a>, <a href="https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/YARN.html">Apache Hadoop Yarn</a>, Spak running in <a href="https://spark.apache.org/docs/2.4.5/cluster-overview.html">cluster mode</a> or any other of your choice.</p><h3>Conclusion</h3><p>Testing Spark applications can seem more complicated than with other frameworks not only because of the need of preparing a data set but also because of the lack of tools that allow us to automate such tests. By means of the <em>SharedSparkSessionHelper</em> class we can automate our tests in an easy way and it should work smoothly with pytest and <a href="https://docs.python.org/3/library/unittest.html">unittest</a>.</p><p><em>I hope this article was useful. If you enjoy messing around with Big Data, Microservices, reverse engineering or any other computer stuff and want to share your experiences with me, just follow </em><a href="https://twitter.com/gumartinm"><strong><em>me</em></strong></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c2ba71467d85" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>