<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Daniel Paes on Medium]]></title>
        <description><![CDATA[Stories by Daniel Paes on Medium]]></description>
        <link>https://medium.com/@danspaes?source=rss-4e465c109e49------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*ZgfVinIoSmtlswucrkmnNw.png</url>
            <title>Stories by Daniel Paes on Medium</title>
            <link>https://medium.com/@danspaes?source=rss-4e465c109e49------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 10:18:54 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@danspaes/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Protect your data with Microsoft Purview, achieving Azure data governance and enterprise risk…]]></title>
            <link>https://medium.com/coderbyte/protect-your-data-with-microsoft-purview-achieving-azure-data-governance-and-enterprise-risk-7a5fd0538785?source=rss-4e465c109e49------2</link>
            <guid isPermaLink="false">https://medium.com/p/7a5fd0538785</guid>
            <category><![CDATA[data]]></category>
            <category><![CDATA[security]]></category>
            <category><![CDATA[big-data-analytics]]></category>
            <category><![CDATA[azure]]></category>
            <dc:creator><![CDATA[Daniel Paes]]></dc:creator>
            <pubDate>Mon, 02 Oct 2023 12:44:17 GMT</pubDate>
            <atom:updated>2023-10-03T10:28:45.076Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>Protect your data with Microsoft Purview, achieving Azure data governance and enterprise risk compliance.</strong></h3><h4>The components of Microsoft Purview and how it can securely manage your data initiatives.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*a2O-22yfKZICvrRV" /><figcaption>Photo by <a href="https://unsplash.com/@sajadnori?utm_source=medium&amp;utm_medium=referral">Sajad Nori</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Using data to have competitive differences has become crucial in some business domains, and thanks to technology advancements, it is now possible to get data from sources and use formats that weren’t around a couple of years ago. However, with these new ingestions, there is a need for updated procedures and tools to achieve Data governance. This publication will explore how Azure’s Purview allows your team to achieve robust data governance in your organization. We will start by checking some of its most used components and understanding the core concepts and tool functionalities, such as data maps and how it composes the data catalog component. Once we get that covered, I’d like to present some valuable considerations to be taken alongside good practices when managing your Microsoft purview environment. Let’s start by revisiting the Microsoft Purview solutions and their components.</p><h3>Overlook on Microsoft Purview solutions and its components.</h3><p>The Microsoft purview suite counts with two solutions, targeting different use cases; the <a href="https://learn.microsoft.com/en-us/purview/governance-solutions-overview">unified data solutions</a> and the <a href="https://learn.microsoft.com/en-us/purview/purview-compliance">risk governance solutions</a>. And while the unified data solutions aim to facilitate the onboarding and exploration of new data sources whilst updating the existing ones. The risk governance solutions count with out-of-the-box classifiers that prevent data from being wrongly shared. However, it also allows you to create custom classifiers to fit your business data compliance better. Both services, alongside the Synapses analytics, build up a solid tool shed when designing and developing your data governance implementation. Let’s start by checking the components of the unified data solutions options under the Microsoft Purview governance portal.</p><h3>Microsoft Purview unified data solution governance portal.</h3><p>Unified data solutions are data services that help data producers, data consumers, and data officers to know which data assets exist in the organization. It is based on the open-source project Apache Atlas being accessible by the user interface from the purview governance portal or over the REST APIs available for Apache Atlas. It also allows the sharing of data securely in an optimized way, needing only to configure workflows to guardrail the approbation pipeline when sharing data. The schema below represents how the Microsoft purview governance portal is structured.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*CyQuRgmskfBONl9X" /><figcaption>Microsoft purview governance portal</figcaption></figure><p>The Microsoft purview unified data solution components bring valuable outcomes when it comes to:</p><ul><li>Data classification and labeling</li><li>Business Glossary</li><li>Lineage Extraction</li></ul><p>Let’s start by understanding what a data map is, and its crucial role on Microsoft purview.</p><h3>What is a Data map?</h3><p>The data map is the building block for unified data governance solutions. It scans the metadata from different sources, such as operational systems from on-premises or another cloud provider, for organizations with multi-cloud environments.</p><p>Each data map starts with one capacity unit, which is auto-scaled to fit the necessary compute, having a base charge of 1 capacity unity. Each capacity unity allows 10 GB of metadata storage and 25 operations per second. The logical aggregation of different data maps is called collections. Each Data map has operations and capacity unity.</p><p>Below is an example of a data map with <strong><em>corsoPurview </em></strong>as root collection while having <strong><em>Outreach, Development, Sales, Marketing, Finance, Azure data lake, and powerBI datasets </em></strong>as leaf collections.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*YMQ2fmsFtRZI5v_L" /></figure><p>Different authentication methods are required when configuring each data source, Microsoft purview counts with the following ones:</p><ul><li>Managed Identity</li><li>Service Principal</li><li>SQL Authentication</li><li>Windows Authentication</li><li>Role ARN</li><li>Delegated Authentication</li><li>Consumer Key</li><li>Account Key or Basic Authentication</li></ul><p>Now that we’re more clear about what a data map is, we can check on the other components. Starting by the Data catalog.</p><h3>Data catalog</h3><p>The data catalog in Purview was defined at its core to work as a centralized source of truth. Microsoft purview data catalog allows the onboarding of new data assets by creating a Data map. The designed collections will be shown as data assets. The data assets and the business rules, called workflows, allow users to curate and share the data securely, enabling the self-service consumption of your data assets. Workflow contains connectors that orchestrate the necessary actions to validate the data asset. Each data collection will have workflows attached to it, and the parent workflow will be taken in case its leaf data map doesn&#39;t have one in place. It is advantageous when defining enterprise-conformed self-service data access default workflows. The collections are hierarchically structured; in this sense, all configuration and workflow request pipelines on the root or higher hierarchical level are also accessible by their children.</p><p>The data discovery capability of scanning file servers and databases, on-premises and in the cloud, is very handful. Microsoft purview data discovery allows data to be reviewed on different network topologies, such as found on-prem, alongside being able to run under Vnet subnets for an extra layer of security.</p><h3>Data Policy</h3><p>The data policy option becomes available by enabling the Policy enforcement on the Microsoft Purview admin console. The data policy statement lists allowed access actions per data resource per subject, allowing hierarchical enforcement of policies, making it possible to deny data services on a higher level with granular permission to its data map collection leaves.</p><h3>Data estate insights</h3><p>Microsoft purview gives a bird’s-eye view of your data assets. The data estate insights is a robust data stewardship tool that allows data stakeholders to have a unified view that facilitates the data management, compliance, and usage by its data stewards. It provides valuable data about governance gaps by highlighting them in its top metrics.</p><h3>Best practices when using Data governance solutions in Microsoft&#39;s Purview</h3><p>Now that we have covered each listed component, let&#39;s revisit some hints when using and configuring your data governance solutions in Microsoft purview.</p><p>Your data stewards can be notified of real-time changes in the Microsoft purview environment and take immediate action. Thanks to Kafka notifications and Azure event hub. The interaction is done using the Microsoft purview account. By this writing, the Kafka topic created by Microsoft purview can have the following statuses.</p><pre>ENTITY_CREATE            : create an entity. <br>ENTITY_FULL_UPDATE       : update an entity. <br>ENTITY_PARTIAL_UPDATE    : update specific attributes of an entity. <br>ENTITY_DELETE            : delete an entity. <br>ENTITY_CREATE_V2         : create an entity. <br>ENTITY_FULL_UPDATE_V2    : update an entity. <br>ENTITY_PARTIAL_UPDATE_V2 : update specific attributes of an entity. <br>ENTITY_DELETE_V2         : delete one or more entities.</pre><p>Managed identities are the preferred authentication method when interacting with Purview. Extra layers of security can be achieved by using private endpoints configured on Azure-secured subnets or VNETS. It allows secure endpoints to be used by Purview to access its portal, integrate with Azure data services, and for Purview account management.</p><p>By designing the scope of your scan, your team will lower data leaks due to broad read permissions. It is good to note that the account used when exploring the data sources must have Data Reader access authorization configured.</p><p>Implementing the least privilege principle while creating the Data policy is also a good practice. It is an excellent practice to have different access policies to handle your user principals and groups, another to take the self-service workflow, and one last to manage your DevOps pipelines. Then, creating custom policies per organization requirements narrows your attack surface due to credential loss. More details about best practices while building and maintaining Azure environments can be found on <a href="https://learn.microsoft.com/en-us/azure/well-architected/">Microsoft Azure Well-architected Framework</a>.</p><p>Now that we have covered the components of data governance of Microsoft purview solutions. Let&#39;s discuss some positive and negative aspects of using Microsoft purview solutions for data governance.</p><h3>The Good</h3><p>The onboarding of new data sources and the capability of grouping them into logical components, called collections, are very useful. Allowing to expand not only on data center servers where the data resides but also per department, grouping all data services they use.</p><p>Also, the lineage is not limited to database entities such as tables and views, but it also allows you to integrate temporary data containers created by your ingestion code. Microsoft purview enables the exploration of temporary data containers used by Azure Data Factory, Synapse SQL entities, and data visualization like PowerBI and Looker. Allowing the cataloging of sub artifacts such as datasets and tables in addition to Azure Data Factory and Azure Synapse Spark jobs, Microsoft purview can also scan single CSV zipped files in GZIP format.</p><p>The support for stored procedures in Snowflake and code stored in Azure Databricks is also a relief for some departments where most of their analytics runs. It is also comforting for some users to support modeling tools such as Erwin, adding support for legacy systems that still count on it. It also supports other cloud provider services, such as Google cloud storage and Amazon S3.</p><h3>The Bad</h3><p>Microsoft purview is a solid data governance platform. However, there are some flaws like any other tool. One bad aspect is the regional limitation of the Azure purview account coverage when scanning and onboarding new data sources. While it helps secure the data, it should allow managing multiple regions on a centralized platform. Creating multiple Microsoft purview accounts, one per Azure region, where your data resides, is suggested as security best practice.</p><p>It is good to remember that additional accounts will lead to higher costs as the computer cannot be shared. The compute configuration is settled during the purview account creation alongside the region the report will cover.</p><p>Another down factor is that sharing the compute used by any Azure Data Factory self-hosted integration runtime with Microsoft Purview self-hosted integration runtime is impossible.</p><h3>The Ugly​</h3><p>Unfortunately, the support for MDM doesn’t come as a feature of Microsoft Purview by the time of this writing. To achieve Master Data Management capabilities, a partner solution developed by Profisee is needed. When used in conjunction, the Microsoft Purview adds up to the MDM functionality, with the cost of adding an extra service that needs to be managed in addition to your Microsoft Purview environment.</p><p>While using Microsoft Purview, there are some valuable hints that we list. Check the best practices when using data governance solutions under Microsoft&#39;s Purview.</p><p>Now let’s check on what Microsoft Purview Risk and Compliance solutions are.</p><h3>Microsoft Purview Risk and Compliance solutions</h3><p>Microsoft Purview compliance solutions have out-of-the-box classifiers that manage insider risks, information protection, and data lifecycle based on resources tags for conformity reasons. By adding rules, it implements data loss protection by requesting confirmation on some actions, such as confirming sending an email with data objects classified as confidential. It easily integrates with your organization&#39;s Azure Active Directory. Support for Azure AD administrative units is attained by using adaptive scopes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/525/0*-mr8odf3VT57C3QU.png" /></figure><p>Microsoft purview Risk and Compliance solutions enforce guardrails to your data assets, and thanks to the Microsoft purview auditing solutions, your environment becomes more responsive to security events. The auditing solutions provide valuable insights also for root cause analysis of post-mortem events. In addition to the following</p><ul><li>Communication compliance</li><li>Insider risk management</li><li>Information Barriers</li><li>Privileged access management</li><li>Data classification</li><li>Sensitivity labels</li><li>Data lifecycle retention policies</li><li>Data archival and deletion lifecycle policies</li><li>Industry-specific guidance on security and compliance for Financial and Energy services</li></ul><p>Now that we have both significant domains that Microsoft Purview aims to solve. Let&#39;s sum up what we covered in this publication.</p><h3>Conclusion</h3><p>Microsoft purview solutions are a solid choice to have compliance and risk guardrails while providing superior support when managing and onboarding new data into your data governance and observability environments. Based on Apache Atlas, management, and compliance for Hadoop environments, it allows prescriptive and forensic models to operationalize your data sources not limited to Hadoop.</p><p>See you folks next time!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7a5fd0538785" width="1" height="1" alt=""><hr><p><a href="https://medium.com/coderbyte/protect-your-data-with-microsoft-purview-achieving-azure-data-governance-and-enterprise-risk-7a5fd0538785">Protect your data with Microsoft Purview, achieving Azure data governance and enterprise risk…</a> was originally published in <a href="https://medium.com/coderbyte">Tech x Talent</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Cost-effective compliance and security of your data mesh initiatives using DevSecOps]]></title>
            <link>https://danspaes.medium.com/cost-effective-compliance-and-security-of-your-data-mesh-initiatives-using-devsecops-f433ff12e11b?source=rss-4e465c109e49------2</link>
            <guid isPermaLink="false">https://medium.com/p/f433ff12e11b</guid>
            <category><![CDATA[security]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[business-intelligence]]></category>
            <dc:creator><![CDATA[Daniel Paes]]></dc:creator>
            <pubDate>Wed, 06 Sep 2023 01:52:52 GMT</pubDate>
            <atom:updated>2023-10-06T02:07:10.090Z</atom:updated>
            <content:encoded><![CDATA[<h4>Check data pipeline code compliance and security in the early development stages of your data product with no cost implied.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*oLZahvaDcsw7nl86" /><figcaption>Photo by <a href="https://unsplash.com/@r3dmax?utm_source=medium&amp;utm_medium=referral">Jonatan Pie</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Consistent data builds up the competitive differentiator among businesses. Extracting insights from existing data is already a challenge; the challenge intensifies when new data becomes available without the guardrails to onboard these new data sources.</p><p>This publication will discuss the main data mesh principles and types of data it handles. We will revisit the DevSecOps phases and components, and how to secure and develop an enterprise conformed Data products on data mesh projects.</p><p>We will then learn about pre-commit hooks and how they can prevent secret leaks. First, Let’s start by refreshing the data mesh principles and its kinds of data.</p><h3>What are the data mesh principles and its kinds of data?</h3><p>Data mesh answers how to onboard new data thanks to a <strong>decentralized domain ownership</strong> model, a concept with its core in the domain-driven design. Each business domain will share data based on data contracts, as the <strong>data as a product</strong> principle states. It provides autonomy, as each business domain will count on their subject matter experts to create its data pipelines.</p><p>Data pipelines are the common name to describe the data transformations and flows from its source, or data producers, and all the necessary enrichment until it becomes available to its consumers. The subject matter experts will enrich the pipelines with any additional data needed, defining the functional requirements and any extra non-technical characteristics, such as service-level agreements about the compliance and availability of shared data.</p><p>Data mesh presents a shift in how data is processed and maintained within an organization for operational and analytical needs. By providing autonomy while maintaining and authoring new data products, the data mesh <strong>self-serve data platform</strong> principle helps create security and compliance standards by automating federated rules per domain.</p><p>The <strong>federated computational governance </strong>principle aims to reduce the overhead of manual synchronization between the business domains while auditing the compatibility inter-domains while generating the data products. Now that we remembered the data mesh principles, let’s revisit the <strong>Data types found on data mesh initiatives</strong></p><h3>Data types found on data mesh initiatives</h3><p>Data mesh categorize data as either <strong>operational</strong> or <strong>analytical</strong>. Operational data, also known as Online transactional platform or OLTP is the data used by the core applications. For this reason, it should avoid being transformed, risking increasing application latency and being more error-prone due to tampered data.</p><p>Analytical data, on the other hand, is used when manipulation and enrichment are needed to get insights from it. An online analytical platform or OLAP derives the operational data into data used on Key process indicators.</p><p>Now that we have covered the data mesh principles let’s see <strong>the DevSecOps development phases</strong> and how to configure global policies while developing data products.</p><h3>The DevSecOps development phases.</h3><p>DevSecOps stands for <strong>development/security/operations</strong>, achieved by securing all phases of the software development lifecycle. The steps are similar to the ones described in DevOps implementations.</p><ul><li>Plan</li><li>Code</li><li>Build</li><li>Test</li><li>Release</li><li>Deploy</li></ul><p>In addition to the DevOps phases, DevSecOps initiatives count on extra components to handle the code analysis, compliance management, and threat modeling, to name some. While there is a value added by automating some of these checks to the data pipeline, it is not uncommon to have the components listed here by using manual procedures.</p><p>We can get flooded with multiple actions to increase development security and compliance at every stage. However, the compliance and security proposed in this publication target the <strong>code </strong>and <strong>build</strong> phases. With that in mind, let’s learn more about the DevSecOps components applied in these phases that we will cover in this publication. The chosen tools focus on Python code for simplicity; add extra libraries if you use other programming languages.</p><p><strong>Static Application Security Testing (SAST)</strong> checks for known vulnerabilities in the code before pushing it into the git repository; we will count on Bandit to do it for us. And thanks to the <strong>Software Composition Analysis (SCA)</strong>, we will reinforce code quality and compliance. The proposed code will check the file formatting while adding the code page.</p><p>Now that we have covered our bases, we can move on to securing your data products’ development environment; let’s see <strong>data loss prevention of access tokens in the code and build phases</strong>.</p><h3>Data Loss Prevention of access tokens in the code and build phase</h3><p>Finding Hard-coded values on code can help understand the logic behind it in some cases; however, sharing the secrets of your cloud environment can turn out very costly. We will count on pre-commit hooks to avoid secret leakage in addition to running SAST analysis on an airflow dag code. The original code can be found at this <a href="https://github.com/sbi-pdan/dq-adf-airflow-snowflake">GitHub repository</a>, and here is the <a href="https://danielpaes.ca/running-data-quality-on-azure-data-factory-with-airflow-fc2faa4a3c02">original post</a> where I present how to run Data Quality checks on Azure.</p><p>The command below will create a virtual environment for Python called <strong>pchook_venv</strong> and install the pre-commit library. Don’t forget to add the <strong>pchook_venv </strong>on your <strong>.gitignore</strong> file to avoid pushing it into your repository.</p><pre>python3 -m venv pchook_venv<br>source pchook_venv/bin/activate<br>pip install pre-commit</pre><p>Once installed, it will need the configuration file with the desired policies to be validated. The code below will check for secrets while formatting JSON, YAML, and Python files while running static application security testing (SAST) on Python code.</p><pre>---<br>repos:<br>    - repo: https://github.com/pre-commit/pre-commit-hooks<br>      rev: v4.4.0<br>      hooks:<br>          - id: trailing-whitespace<br>          - id: check-added-large-files<br>          - id: end-of-file-fixer<br>          - id: check-yaml<br>          - id: check-json<br>          - id: detect-private-key<br>    - repo: https://github.com/Yelp/detect-secrets<br>      rev: v1.4.0<br>      hooks:<br>          - id: detect-secrets<br>    - repo: https://github.com/nbQA-dev/nbQA<br>      rev: 1.7.0<br>      hooks:<br>          - id: nbqa-pyupgrade<br>          - id: nbqa-mypy<br>          - id: nbqa-flake8<br>          - id: nbqa-pydocstyle<br>          - id: nbqa-check-ast<br>          - id: nbqa-pyupgrade<br>          - id: nbqa-flake8<br>    - repo: https://github.com/pre-commit/mirrors-autopep8<br>      rev: v2.0.4<br>      hooks:<br>          - id: autopep8<br>    - repo: https://github.com/pre-commit/mirrors-mypy<br>      rev: v1.5.1<br>      hooks:<br>          - id: mypy<br>    - repo: https://github.com/frnmst/md-toc<br>      rev: 8.2.0<br>      hooks:<br>          - id: md-toc<br>    - repo: https://github.com/peterdemin/pip-compile-multi<br>      rev: v2.6.3<br>      hooks:<br>          - id: pip-compile-multi-verify<br>    - repo: https://github.com/asottile/pyupgrade<br>      rev: v3.10.1<br>      hooks:<br>          - id: pyupgrade<br>    - repo: https://github.com/asottile/yesqa<br>      rev: v1.5.0<br>      hooks:<br>          - id: yesqa<br>    - repo: https://github.com/PyCQA/flake8<br>      rev: 6.1.0<br>      hooks:<br>          - id: flake8<br>    - repo: https://github.com/PyCQA/bandit<br>      rev: 1.7.5<br>      hooks:<br>          - id: bandit</pre><p>Leave the above code on the root path of your git repository named <strong>.pre-commit-config.yaml.</strong></p><p>The <a href="https://pre-commit.com/#plugins">following link</a> describes how to customize your pre-commit configurations. Once the setup is in place, run the code below for the first check on your repository.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/920/1*BoqZV_6bWMKgvyZNmbP5hg.png" /><figcaption>pre-commit hook checks</figcaption></figure><p>Congratulations! The configured checks on your <strong>.pre-commit-config.yaml</strong> will reinforce the federated compliance policy used on your data mesh initiatives. It reduces your attack surface by avoiding the possibility of cost increase due to credential leakage and reducing the attack surface of your development environment. The proposed pre-commit policy won’t allow compromised commits on your git history. We can now summarize what we covered in this publication while suggesting actions on different software development lifecycle phases to increase your data pipeline security and compliance.</p><h3>Conclusion</h3><p>This publication presents how to implement security in the early stages of development. The proposed approach lowers security risks and threat surfaces; thanks to techniques such as software composition analysis and static application security testing. At the same time, your infrastructure will reinforce compliance policies for your data products.</p><p>Pre-commit also allows custom development that better fits your data contracts and business needs in the <strong>code</strong> and <strong>build </strong>phases. It is good to explore extra components to increase the security of your data pipelines. Adding other DevSecOps components, such as Interactive Application Security Testing (IAST), Dynamic Application Security Testing (DAST), and Runtime application self-protection (RASP), will make your threat modeling model more accurate.</p><p>Implementing the binary authorization model policies in the development stage and running software composition analysis against NIST databases. And enforcing federated compliance among your domains while allowing your team to deploy faster while following security best practices; with simple measures like those described in this publication. Your development team will provide faster releases while maintaining quality and following security best practices, saving time and costs when developing your data products; fostering new data driven initiatives with the correct security approach.</p><p>See you guys next time!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f433ff12e11b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Running data quality checks on Snowflake storage using Azure Data Factory with Airflow]]></title>
            <link>https://danspaes.medium.com/running-data-quality-on-azure-data-factory-with-airflow-fc2faa4a3c02?source=rss-4e465c109e49------2</link>
            <guid isPermaLink="false">https://medium.com/p/fc2faa4a3c02</guid>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[data-quality]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[azure]]></category>
            <dc:creator><![CDATA[Daniel Paes]]></dc:creator>
            <pubDate>Fri, 25 Aug 2023 17:22:36 GMT</pubDate>
            <atom:updated>2023-10-06T02:07:54.925Z</atom:updated>
            <content:encoded><![CDATA[<h4>Building blocks on how to trust more your own data</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*5R8xdO1hB0SkrNnM" /><figcaption>Photo by <a href="https://unsplash.com/@mikofilm?utm_source=medium&amp;utm_medium=referral">Mike Kononov</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Most companies need to adapt over the years to keep their business competitive, and having good data plays a crucial role in its success. It is not uncommon that analysts and developers must access their data without any handling, sometimes accessing the application database to avoid extra development steps. However, data quality steps aid by adding helpful validation steps and will give you spare time to analyze without adding latency to your application.</p><p>This post shows the commonly used data quality disciplines and how they can help you while analyzing your data. Using <a href="https://azure.microsoft.com/en-ca/products/data-factory/?ef_id=_k_EAIaIQobChMI3-G_x5j4gAMVaEp_AB2VRgbMEAAYASAAEgIn8_D_BwE_k_&amp;OCID=AIDcmmqz3gd78m_SEM__k_EAIaIQobChMI3-G_x5j4gAMVaEp_AB2VRgbMEAAYASAAEgIn8_D_BwE_k_&amp;gad=1&amp;gclid=EAIaIQobChMI3-G_x5j4gAMVaEp_AB2VRgbMEAAYASAAEgIn8_D_BwE">Azure Data Factory</a>, we will see how to audit your data quality jobs on the Airflow integration runtime, validating data stored on Snowflake. We will conclude by circling back to what we covered and presenting items to consider when choosing Airflow as your integration runtime. Check our GitHub repository to find the proposed code in more detail at <a href="https://github.com/sbi-pdan/dq-adf-airflow-snowflake">https://github.com/sbi-pdan/dq-adf-airflow-snowflake</a>.</p><p>Let’s start by checking what Data Quality is and its dimensions.</p><h3>What is data quality and its dimensions?</h3><p>Data quality, also known as Data Quality and observability, is one of the pillars of <a href="https://www.techtarget.com/searchdatamanagement/definition/data-governance">data governance</a>. It consists of processes and data pipelines responsible for cleaning, auditing, and modelling to fit the business requirements. It consists not only of the business rules to load the data but also extra constraints, such as not handling data that does not fit on a service level agreement for that data product; or even having inconsistent data available over time.</p><p>Overall, data quality projects consist of the <a href="https://www.techtarget.com/searchdatamanagement/tip/6-dimensions-of-data-quality-boost-data-performance">following disciplines</a>. And, despite having enterprise-conformed procedures for different domains, it is good to reinforce that each data pipeline should count with individual data quality processes when possible.</p><ul><li>Accuracy</li><li>Completeness</li><li>Consistency</li><li>Validity</li><li>Uniqueness</li><li>Integrity</li></ul><h4>Accuracy</h4><p>It reinforces the accuracy of third-party data used on your data pipeline compared to the source of the information. The accuracy discipline confirms that the data used is accurate and matches the source system. A typical example is to validate financial data against physical tax files to validate the deductible taxes.</p><h4>Consistency</h4><p>It states how consistent your data is within your ecosystem. The consistency discipline confirms if the attribute has the same value among different data sources. An example would be to check the digital version of documents must match their original paper one.</p><h4>Completeness</h4><p>It underlines how to complete your data from what was available. The completeness discipline does not include having all fields filled up from the original, as your data can have optional columns. An example is having all the data available within the defined service level agreement for the data product.</p><h4>Integrity</h4><p>It builds up that you don’t have tampered data in your ecosystem. The integrity discipline confirms that the data used on your data pipelines was processed as expected from its source, either third-party or as a result of a business transformation. An example would be to block ad-hoc or manual changes on the data where it is stored and permanent changes to be made by the data pipeline that loads the table only.</p><h4>Uniqueness</h4><p>It narrows the data duplication from disparate sources from overlapping data. The uniqueness discipline reinforces that each data pipeline handles non-duplicated datasets within your environment. An example could be having decision points to avoid the same data loaded multiple times.</p><h4>Validity</h4><p>It supports that the data is in the expected format to be used by the business. The validity discipline is not necessarily only technical, such as data types; it confirms that the data has values that match the business expectations. An example is configuring the data pipeline to verify that name fields are not only numbers.</p><p>Now that we covered the basics let’s see it in action. First, let’s start by creating our Azure Data Factory integration runtime environment.</p><h3>Create the Airflow integration runtime on Azure Data Factory.</h3><p>First, you need to create an Azure Data factory in the Azure region where you will run your code. Our work will be on the US region, as Airflow is currently unavailable to all Azure regions. Airflow runtimes will be available on the “<strong>workflow orchestration manager” </strong>page under the <strong>manage</strong> option on your Azure Data Factory homepage.</p><p>We will configure the snowflake configuration as an environment variable in Airflow, as it increases the security of your environment. The code below creates a new <strong><em>“snowflake_default” </em></strong>Airflow connection; please change the below link to match your Snowflake environment. We will use it to connect to Snowflake, and it is also necessary to inform <strong>apache-airflow-providers-snowflake </strong>on the <strong><em>airflow requirements </em></strong>field.</p><pre>AIRFLOW_CONN_SNOWFLAKE_DEFAULT=snowflake://&lt;Snowflake user&gt;:&lt;Snowflake pwd&gt;@/&lt;Snowflake Schema&gt;?account=&lt;Snowflake account&gt;&amp;database=&lt;Snowflake Database&gt;&amp;region=&lt;The Cloud provider region&gt;&amp;warehouse=&lt;Snowflake warehouse&gt;</pre><p>There are different ways to import your code into Airflow. Today, we will use the git integration to update the integration runtime automatically. We will use the code at the main branch on <a href="https://github.com/sbi-pdan/azdf-aifl-snfk">https://github.com/sbi-pdan/azdf-aifl-snfk</a> for our example. Below is an example of a valid configuration.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/634/1*Hqsbeyy1ejm41WJnBZRmcA.png" /><figcaption><em>Airflow environment setup</em></figcaption></figure><p>Now that our environment is ready let’s see how to run your job and audit it using the Airflow user interface.</p><h3>Trigger and audit your Airflow data quality pipeline on the Airflow UI.</h3><p>Below is the code that runs data quality checks on a Snowflake warehouse using Airflow.</p><pre>from datetime import datetime<br>from airflow.models import DAG<br>from airflow.operators.dummy import DummyOperator<br>from airflow.providers.common.sql.operators.sql import (<br>    SQLColumnCheckOperator,<br>    SQLTableCheckOperator,<br>)<br><br>with DAG(<br>    dag_id=&quot;snowflake_airflow_demo&quot;,<br>    start_date=datetime(2023, 8, 1),<br>    schedule_interval=None,<br>    catchup=False,<br>    default_args={<br>        &quot;conn_id&quot;: &quot;snowflake_default&quot;,<br>    },<br>) as dag:<br>    start = DummyOperator(task_id=&quot;start&quot;)<br>    end = DummyOperator(task_id=&quot;end&quot;)<br><br># • Completeness measures how comprehensive your data is.<br>    &quot;&quot;&quot;<br>    #### Completeness<br>    &quot;&quot;&quot;<br>    supplier_completeness = SQLColumnCheckOperator(<br>        task_id=&quot;completeness&quot;,<br>        table=&quot;supplier&quot;,<br>        column_mapping={<br>            &quot;S_SUPPKEY&quot;: {&quot;null_check&quot;: {&quot;equal_to&quot;: 0}},<br>            &quot;S_NAME&quot;: {&quot;null_check&quot;: {&quot;equal_to&quot;: 0}},<br>            &quot;S_ACCTBAL&quot;: {&quot;null_check&quot;: {&quot;equal_to&quot;: 0}},<br>            &quot;S_NATIONKEY&quot;: {&quot;null_check&quot;: {&quot;equal_to&quot;: 0}},<br>        },<br>    )<br><br># • Accuracy measures how correct your data is in representing objects within the scope of requirements.<br>    &quot;&quot;&quot;<br>    #### Accuracy<br>    &quot;&quot;&quot;<br>    supplier_accuracy = SQLColumnCheckOperator(<br>        task_id=&quot;accuracy&quot;,<br>        table=&quot;supplier&quot;,<br>        column_mapping={<br>            &quot;S_ACCTBAL&quot;: {&quot;min&quot;: {&quot;geq_to&quot;: -1000}},<br>            &quot;S_ACCTBAL&quot;: {&quot;max&quot;: {&quot;leq_to&quot;: 10000}},<br>            &quot;S_NATIONKEY&quot;: {&quot;min&quot;: {&quot;geq_to&quot;: 0}},<br>            &quot;S_NATIONKEY&quot;: {&quot;max&quot;: {&quot;leq_to&quot;: 50}},<br>        },<br>    )<br><br># • Uniqueness looks at duplication within your data.<br>    &quot;&quot;&quot;<br>    #### Uniqueness<br>    &quot;&quot;&quot;    <br>    supplier_uniqueness = SQLColumnCheckOperator(<br>        task_id=&quot;uniqueness&quot;,<br>        table=&quot;supplier&quot;,<br>        column_mapping={<br>            &quot;S_SUPPKEY&quot;: {&quot;unique_check&quot;: {&quot;equal_to&quot;: 0}},<br>        },<br>    )<br><br># • Timeliness ensures that your data is fresh and available within your requirements/SLAs.<br>    &quot;&quot;&quot;<br>    Timeliness<br>    &quot;&quot;&quot;<br>    supplier_timeliness = SQLTableCheckOperator(<br>        task_id=&quot;timeliness&quot;,<br>        table=&quot;supplier&quot;,<br>        checks={&quot;row_count_check&quot;: {&quot;check_statement&quot;: &quot;COUNT(*) = 10000&quot;}},<br>    )<br><br>start  &gt;&gt; supplier_completeness &gt;&gt; supplier_accuracy &gt;&gt; supplier_uniqueness &gt;&gt; supplier_timeliness &gt;&gt; end</pre><p>To update the airflow environment, the developer must commit new code into the GitHub branch and the repository configured on Azure Data Factory. The code will be shortly available on the Airflow DAG page.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ilPpVUkrl6XRGdhTvpdkIA.png" /><figcaption><em>Airflow User Interface</em></figcaption></figure><p>One can find on this endpoint extra information, such as the previous runs and the source code of your pipeline; it also gives the option to start your airflow pipeline ad hoc. You’ll need to click the “trigger” button to do so.</p><p>Azure Data Factory sends all integration runtime events to Azure monitor for further analysis. However, we can also count on the user interface from Airflow to help while auditing our code. By clicking on the dag name on the Airflow UI, more details are available, such as the logs of each airflow task. Below is an example of a successful data quality checking the completeness.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2KF1CxjggoSwaGmAcBXMfw.png" /><figcaption><em>Successful completeness data quality check</em></figcaption></figure><p>Now that we have learned how to audit our data pipeline, we can summarize what we covered with some extra things to consider when using Apache Airflow under Azure Data Factory.</p><h3>Conclusion and things to consider</h3><p>In this publication, we saw how to improve the quality of your data stored on Snowflake by running pipelines using Azure Data Factory. We saw how to configure Airflow as an integration runtime and how to run and audit data quality checks.</p><p>Data quality checks are essential for robust data governance. However, the question answered plays a crucial role. It gives nothing to have data observability and monitoring in place for your data pipelines to infer better data for irrelevant questions to your line of business.</p><p>It is good to note that Airflow is unavailable on all Azure regions, which can incur extra traffic charges when ingesting data from different Azure regions. In addition, integration with Microsoft services such as <a href="https://www.microsoft.com/en-ca/security/business/microsoft-purview">Microsoft purview</a> is not available when the integration runtime is Airflow. Some airflow capabilities, such as <a href="https://airflow.apache.org/docs/apache-airflow/1.10.12/profiling.html">data profiling</a> using ad-hoc queries in your data sources, are also unavailable.</p><p>See you folks next time.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fc2faa4a3c02" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Tableau Server on Containers]]></title>
            <link>https://medium.com/coderbyte/tableau-server-on-containers-1ecb804a3db5?source=rss-4e465c109e49------2</link>
            <guid isPermaLink="false">https://medium.com/p/1ecb804a3db5</guid>
            <category><![CDATA[analytics]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[tableau]]></category>
            <category><![CDATA[docker]]></category>
            <dc:creator><![CDATA[Daniel Paes]]></dc:creator>
            <pubDate>Wed, 22 Dec 2021 16:11:50 GMT</pubDate>
            <atom:updated>2021-12-25T13:56:18.834Z</atom:updated>
            <content:encoded><![CDATA[<h4>Should you bother deploying it this way in the end?</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*4xgtXsRnBi0yZSlz" /><figcaption>Photo by <a href="https://unsplash.com/@d_mccullough?utm_source=medium&amp;utm_medium=referral">Daniel McCullough</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>It is a recurrent challenge to present your data in an appealing format that gets the importance that is needed. And today, I want to explore more of the big names of data visualization, the tool of love and hate for those who are long enough on the data analytics; <a href="https://www.tableau.com/">Tableau</a>.</p><p>As refactoring existing systems from monolithic as microservices. It would be a matter of time that the Tableau folks provided a recipe to help your company to achieve it not that long ago. But before we jump in, some might want to refresh some of what is microservices. <a href="https://danielpaes.ca/making-some-sense-from-microservices-and-monolithic-architectures-a-brief-storytelling-1caf2c16aa49">This link</a> is an excellent refresher on what microservice is, and <a href="https://www.tableau.com/about/blog/2021/6/deploy-tableau-server-linux-docker-container">here</a>, you can find the code deployment from Tableau for your experiments; and you might also want to get an overall idea of <a href="https://kubernetes.io/docs/concepts/overview/what-is-kubernetes/">Kubernetes</a> using <a href="https://docs.docker.com/engine/reference/commandline/images/">docker container images</a> as implicitly they will be referenced.</p><p>Now that we have that covered, we can discuss the good and not-so-good surprises when deploying Tableau using microservices. We will end up with a quick overview of what is for you if you refactor its deployment from monolithic into microservices, so let’s get started.</p><h4>It is container-ish . . .</h4><p>As Tableau has some internal processes, it is necessary to have a process control system to have it deployed correctly. For this reason, Tableau cannot be deployed using slimmer initialization processes. Talking on practical terms, you cannot use <em>systemd </em>to deploy your containers, the <em>supervisord </em>is used instead.</p><p>Supervisord is a great tool that could be said that be a process control system like Kubernetes. So in the end your containers will be smaller orchestrated processes inside an orchestration tool.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*BbWKWVZJCAe88b2R" /><figcaption>Virtual machines inside containers — Inception!?! (Photo by <a href="https://unsplash.com/@theregisti?utm_source=medium&amp;utm_medium=referral">TheRegisti</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a>)</figcaption></figure><p>It comes without saying that it turns your Kubernetes deployment into a Virtual Machine management tool like VirtualBox. Nothing wrong here (even tho I would use Vagrant instead of Kubernetes), but if the end state will need to use another type of control plane so Kubernetes here might seem unnecessary.</p><p>There is still the possibility of deploying it as another service on your pod, but have in mind that your cluster might run out of capacity if Tableau gets hungry. You can still have a workaround having your High Availability in place with <a href="https://cloud.google.com/architecture/service-meshes-in-microservices-architecture">Service mesh</a> services like <a href="https://istio.io/">Istio</a> to manage the ingress/egress traffic as any Kubernetes deployment, but you would need to have special configurations as your team would use more Tableau for new metrics and dashboards.</p><p>Below you can see which services are deployed alongside your Tableau Server deployment.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*d4olrm0nYsRt0NeO.png" /></figure><h4>It is deployed on Linux versions with heavy community support; it doesn&#39;t support Kerberos though.</h4><p>As it is mentioned on their <a href="https://help.tableau.com/current/server-linux/en-us/server-in-container_setup-tool.htm">server-in-container-tool</a>, you can deploy it on RHEL, CentOS or even <a href="https://linuxhint.com/what_is_amazon_linux_2/">Amazon Linux 2</a> golden image (which is RHEL based anyways). In addition to that, you can also use <a href="https://almalinux.org/">Almalinux</a> for those distributions. This could put the mind at ease for those sysadmins out there who are concerned about the discontinuation of <a href="https://medium.com/keyrus-canada/some-things-do-change-alternative-options-for-centos-and-docker-6d99fb593a3">CentOS</a>.</p><p>Its limitations do not stop there. Deploying Tableau doesn&#39;t provide you Kerberos support too. And to be operational it needs to be able to connect to the public internet to validate if <a href="https://help.tableau.com/current/server-linux/en-us/server-in-container_setup-tool.htm#complete-the-registration-form">your license is still valid</a>.</p><h4>Should you bother deploying it this way in the end?</h4><p>Unless you want to prove a point to use Tableau on containers, I wouldn&#39;t push too much on it. As mentioned, with the Amazon Linux 2 support, it would be preferable to have your EC2 fleets with bootstrap scripts handling all those tangled starting processes that some hate to manage.</p><p>There are some nice addons presented on <a href="https://www.tableau.com/events/aws-reinvent-home-2021-11-29">AWS:Reinvent</a> such as having a better support to get data from S3 in addition of having an automated deployment of Tableau Server. But I wouldn’t put my eggs on this basket yet, let’s sleep over it before getting your production pipeline on it.</p><p>Below you can find a graphical representation of how Tableau should be deployed on AWS ecosystems.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/572/1*lHa_vcR8iUUKRD05T675Cg.png" /></figure><p>Getting back on your container administration tasks; below you can see the output of the build-image script which is used when baking your container images. As we can see, It also helps you with the User acceptance agreement, it stores locally a copy. Another explanation for the need for the supervisord.</p><pre>./src/build-image -h<br>Usage: build-image --accepteula -i [INPUT_IMAGE_NAME] -o [OUTPUT_IMAGE_NAME] [optional arguments]<br>Creates new Tableau Server image with pre-installed database drivers, configuration, and instructions.<br>REQUIRED<br><br>--accepteula              Indicate that you have accepted the End User License Agreement (EULA).<br><br>The EULA is available in the EULA directory of this tool.<br><br>-i installer              Path to the Tableau Server installer.<br><br>OPTIONAL<br><br>-o output name            Tag of target generated Tableau Server image.<br><br>-e environment file       Optional environment file to configuration overrides to docker image at build time.<br><br>-v setup tool version     Prints the Setup tool version.<br><br><br>-b base image version     Prints the default base image URL.</pre><blockquote>build-image output from the Container Setup Tool.</blockquote><p>If your company is not on AWS, no problem! You can have your boilerplate scripts to create your golden images and deploy them on <a href="https://cloud.google.com/compute/docs/images/create-delete-deprecate-private-images">Compute Engine</a> (Google Cloud) or Azure <a href="https://docs.microsoft.com/en-us/azure/virtual-desktop/set-up-golden-image">Virtual machines</a> (Azure) for example. For more comparison between the services please go to this <a href="https://cloud.google.com/free/docs/aws-azure-gcp-service-comparison">link</a>. But it is a small price for what Tableau could provide as insights from your data.</p><p>See you guys next time!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1ecb804a3db5" width="1" height="1" alt=""><hr><p><a href="https://medium.com/coderbyte/tableau-server-on-containers-1ecb804a3db5">Tableau Server on Containers</a> was originally published in <a href="https://medium.com/coderbyte">Tech x Talent</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AWS raising the bar for Data Analytics.]]></title>
            <link>https://medium.com/keyrus-canada/aws-raising-the-bar-for-data-analytics-8eeadeb99501?source=rss-4e465c109e49------2</link>
            <guid isPermaLink="false">https://medium.com/p/8eeadeb99501</guid>
            <category><![CDATA[serverless]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <dc:creator><![CDATA[Daniel Paes]]></dc:creator>
            <pubDate>Wed, 08 Dec 2021 21:29:23 GMT</pubDate>
            <atom:updated>2021-12-08T21:29:23.922Z</atom:updated>
            <content:encoded><![CDATA[<h4>AWS:Reinvent 2021 brought us very tasty serverless options for Redshift</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Qlp0rcq3K1g-Tmyx" /><figcaption>Photo by <a href="https://unsplash.com/@austinchan?utm_source=medium&amp;utm_medium=referral">Austin Chan</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p><a href="https://reinvent.awsevents.com/keynotes/?trk=em_a134p000007CtkcAAC">AWS:Reinvent</a> brought some very juicy updates on some of their offered services for analytics. One remarkable one is with <a href="https://aws.amazon.com/blogs/aws/aws-lake-formation-general-availability-of-cell-level-security-and-governed-tables-with-automatic-compaction/">Lake formation</a>; now, it is possible to reinforce access based on row and attribute level. Not only that, but now it is possible to have ACID properties when interacting with files stored on AWS S3; last but not least, the automatic file compression by AWS Athena and Redshift Spectrum when reading from small files.</p><p>And the news doesn’t stop there for Redshift. In addition to having a <a href="https://aws.amazon.com/blogs/aws/amazon-redshift-query-editor-v2-web-query-authoring/">new free web console</a> to interact with your clusters, Redshift now has a serverless option. This new feature can be a game-changer for some business domains with tons of data, but not that mature IT department.</p><h4>New console UI</h4><p>I like the web UI console to interact with the Redshift cluster. The option to use temporary passwords is a very handful; it is good to see that this option remains, as we can see below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/972/0*UnaJGmUe4b4ra3Ao.png" /><figcaption>Temporary credentials for authentication</figcaption></figure><p>Another good thing with this new version is exporting the query results as charts. Which, in most cases, could be very helpful if you cannot afford presentation tools that need server management. This feature is an add-on that could save you and your team some sleepless nights.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*wJEt1zMgh_un8sm8.png" /><figcaption>Now with the new UI you have the option of creating and exporting charts too.</figcaption></figure><p>One starting point would be to properly configure your permissions (take a look at what is needed <a href="https://docs.aws.amazon.com/redshift/latest/mgmt/redshift-iam-access-control-identity-based.html#redshift-policy-resources.required-permissions.query-editor-v2">here</a>). Another thing to have in mind is that the available quota for its utilization is very interesting too; it becomes another factor that helps its adoption.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4OQ8CWD17FEmDNnrved27w.png" /><figcaption>Quota for web query console version 2.</figcaption></figure><h4>Redshift serverless, very appealing.</h4><p>Having Redshift as a serverless option removes the gap between Redshift and well-known names like <a href="https://www.snowflake.com/">Snowflake</a>. Before Redshift, the only AWS Managed Serverless option very used on analytics that could get close was Aurora due to its relational structure.</p><p>Even <a href="https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_PostgreSQL.S3Import.html#USER_PostgreSQL.S3Import.Overview">importing</a> data stored on <a href="https://aws.amazon.com/s3/">AWS S3</a> into native Aurora managed tables or exploring it with <a href="https://docs.aws.amazon.com/athena/latest/ug/creating-tables.html#creating-tables-how-to">Athena</a> (functionality similar to what <a href="https://docs.aws.amazon.com/redshift/latest/dg/c-getting-started-using-spectrum.html">Redshift Spectrum</a> does); is charged differently. <strong>Redshift Processing Units (RPUs)</strong> follow the same model as <a href="https://aws.amazon.com/glue/pricing/">Data Processing Units</a> from AWS Glue. So you will only pay for your Redshift endpoint compute, being charged by its uptime.</p><p>It is good to point out that you don’t need the web console v2 as your serverless cluster is still accessible with the <a href="https://docs.aws.amazon.com/redshift/latest/mgmt/working-with-serverless.html#serverless-data-api">Redshift Data API</a>. If you haven’t used it until now, you should consider it; doing so will help create independent modules to clean and wrangle your data. Below you can find the reference of the Data API data types from a JDBC connection; it could be helpful while building your own data pipelines.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6dBOjFb8ULktUitrLnMtyQ.png" /><figcaption>Redshift Data API Datatypes reference.</figcaption></figure><p>Does it sound interesting? Here’s an extra for your time then! What if I tell you that you can handle most of your daily loads by “<a href="https://docs.aws.amazon.com/redshift-data/latest/APIReference/API_ExecuteStatement.html">ExecuteStatement</a>” and “<a href="https://docs.aws.amazon.com/redshift-data/latest/APIReference/API_BatchExecuteStatement.html">BatchExecuteStatement</a>” operations. Check <a href="https://docs.aws.amazon.com/redshift//latest/mgmt/data-api.html">here</a> for more details on their configurations requirements and their <a href="https://docs.aws.amazon.com/redshift-data/latest/APIReference/Welcome.html">reference page</a> extra operations.</p><p>As expected, Redshift serverless comes from some limitations on its <a href="https://docs.aws.amazon.com/redshift/latest/mgmt/working-with-serverless.html#serverless-monitoring">monitoring</a> from your clusters. Nothing too heavy for the daily users as you can still have access to essential metadata from SVV_ALL_COLUMNS. Or essential information about the availability of your data between different department divisions. And how the data is shared thanks to SVV_DATASHARES, SVV_DATASHARE_CONSUMERS and SVV_DATASHARE_OBJECTS. But you would like to familiarize yourself with the following ones:</p><ul><li><a href="https://docs.aws.amazon.com/redshift/latest/mgmt/working-with-serverless.html#SYS_QUERY_DETAIL">SYS_QUERY_DETAIL</a></li><li><a href="https://docs.aws.amazon.com/redshift/latest/mgmt/working-with-serverless.html#SYS_EXTERNAL_QUERY_DETAIL">SYS_EXTERNAL_QUERY_DETAIL</a></li><li><a href="https://docs.aws.amazon.com/redshift/latest/mgmt/working-with-serverless.html#SYS_LOAD_HISTORY">SYS_LOAD_HISTORY</a></li><li><a href="https://docs.aws.amazon.com/redshift/latest/mgmt/working-with-serverless.html#SYS_LOAD_ERROR_DETAIL">SYS_LOAD_ERROR_DETAIL</a></li><li><a href="https://docs.aws.amazon.com/redshift/latest/mgmt/working-with-serverless.html#SYS_UNLOAD_HISTORY">SYS_UNLOAD_HISTORY</a></li><li><a href="https://docs.aws.amazon.com/redshift/latest/mgmt/working-with-serverless.html#SYS_SERVERLESS_USAGE">SYS_SERVERLESS_USAGE</a></li></ul><h4>Is it for me?</h4><p>It is always safe to stay on the same provider. Management tasks such as network connectivity could increase the complexity to maintain it within your company. It is not easy to dig into the correct applicability for your use case. It will be a dead-end focus, only the problems that could come from it. Data is already proven to be the biggest asset in <a href="https://www.economist.com/leaders/2017/05/06/the-worlds-most-valuable-resource-is-no-longer-oil-but-data">the new age</a>.</p><p>With that said, it will be a matter of time for your company to either adapt to new ways of handling your data and the steep curve of adopting new technologies. A solid roadmap is mandatory from your IT department. This document must contain all necessary studies for mitigating your technical debt; not building a feasible plan for your reality can be why you perish from not reacting where your company is bleeding, according to your data.</p><p>See you guys next time!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8eeadeb99501" width="1" height="1" alt=""><hr><p><a href="https://medium.com/keyrus-canada/aws-raising-the-bar-for-data-analytics-8eeadeb99501">AWS raising the bar for Data Analytics.</a> was originally published in <a href="https://medium.com/keyrus-canada">Keyrus Canada</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Leveraging RavenDB as NoSql option for Document storage]]></title>
            <link>https://medium.com/codex/leveraging-ravendb-as-nosql-option-for-document-storage-8758b8c0d3cc?source=rss-4e465c109e49------2</link>
            <guid isPermaLink="false">https://medium.com/p/8758b8c0d3cc</guid>
            <category><![CDATA[nosql]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[microservices]]></category>
            <dc:creator><![CDATA[Daniel Paes]]></dc:creator>
            <pubDate>Sat, 10 Jul 2021 23:36:32 GMT</pubDate>
            <atom:updated>2021-07-12T11:58:09.745Z</atom:updated>
            <content:encoded><![CDATA[<h4>Everyone can profit from an open-source DocumentDB option with ACID properties for your landing data layer.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*_DsPu7EmsJlFqLwz" /><figcaption>Photo by <a href="https://unsplash.com/@stereophototyp?utm_source=medium&amp;utm_medium=referral">Sara Kurfeß</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Hey guys, how are you guys doing? Today I want to present <a href="https://ravendb.net/">RavenDB</a>, an open-source option for your <a href="https://www.techopedia.com/definition/4604/on-line-transaction-processing-oltp">OLTP</a> systems of the document type. But before you roll your eyes, skeptic saying: “Yeah, my current choice does that, and even pours me a coffee.”</p><p>I want to reassure you that it’s not a long-term commitment here. And leveraging RavenDB as an option for your NoSql needs could grant you some perks. Such as its native support for <a href="https://searchdatamanagement.techtarget.com/definition/OLAP">OLAP</a> modelling, integration with the major clouds and its <a href="https://www.techopedia.com/definition/23949/atomicity-consistency-isolation-durability-acid-database-management-system">ACID</a> properties, to list some.</p><p>Sounds good? So if you have some spare time, I want to present to you what RavenDB does and why it can be a contender for your OLTP systems; we will then see how to query some data. We will wrap up with some use cases and considerations to have in mind for each of them.</p><h3>What is RavenDB</h3><p>RavenDB is a <a href="https://searchdatamanagement.techtarget.com/definition/NoSQL-Not-Only-SQL">NoSql</a> Document type database with exciting features, such as allowing your team to use SQL to explore your semi-structured data in the same way as you do for your structured data.</p><p>To understand how it helps, let’s do a quick recap on NoSql and Relational databases. Let’s start by remembering that our data is categorized by how its generated, and it can be in the following formats:</p><ul><li>Structured (Relational databases or delimited flat-files);</li><li>Semi-structured (nested structures separated by key-value pairs as shown below);</li><li>Non-Structured (multimedia content like Photos and Videos).</li></ul><figure><img alt="JavaScript Object Notation (JSON)" src="https://cdn-images-1.medium.com/max/1024/1*0jB0VzcqGCg92_rr9GzNVg.png" /><figcaption>JavaScript Object Notation (JSON)</figcaption></figure><p>Sometimes you cannot have the happy path of structured and clean data for our reports, so this is an elegant feature if you think about it. Your analysts will profit from boots on their productivity from simple code.</p><p>Enabling more data exploration with less deployment time can reuse the same logic from existing pipelines into new ones. Allowing them to integrate non-tabular sources in the same way as they do with the tabular ones. Being as tasty as ice cream on a hot day for some analysts.</p><h3>Ok, but why should I consider it as a NoSql option?</h3><p>Let’s go deeper now that we got our bases covered. Let’s first start with the ACID property. This feature is native to Relational databases but not commonly found on NoSql databases as they are under <a href="https://queue.acm.org/detail.cfm?id=1394128">BASE</a> properties.</p><p>Another neat feature of RavenDB is what they call “<a href="https://ravendb.net/why-ravendb/multi-model">Multi-Model Database</a>,” and yeah, you got it right, my good friend; It allows you to create canonical models which are decoupled from your application, as the name suggests.</p><p>In simpler terms, you interact with runtime objects instead of the actual tables. These objects expose your data to you, removing the need to import your semi-structured data or create dumps for your structured data. RavenDB generates a logical layer where you can use a unified language to question your data. The good news is that it is their SQL-like language called RQL, so there is no need for complex proprietary languages.</p><p>The last trick, but for sure not the least, that I want to mention that RavenDB brings as your NoSql database option to the table is native ETL <a href="https://ravendb.net/learn/inside-ravendb-book/reader/4.0/8-sharing-data-and-making-friends-with-etl">integration</a>. It uses JavaScript instead of the SQL-like scripting language that I said before (Raven Query Language, or <a href="https://ravendb.net/learn/inside-ravendb-book/reader/4.0/9-querying-in-ravendb#fn:1">RQL</a> for the close ones), but it still better than nothing.</p><h3>Yeah, right; can I see some data now?</h3><p>We will use the RavenDB sandbox for our tutorial. To follow along, go to this <a href="http://live-test.ravendb.net/studio/index.html#dashboard">link</a> to explore it. As we can see below, the RavenDB interface is not that bad. It shows quickly helpful information such as storage usage and databases available, as shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dU7lO1g5rtVAI79CYMvc_g.png" /><figcaption>Raven Studio Console</figcaption></figure><p>Going under the “Indexes” (second option from top left), you can see the existing indexes upon which you can query your data. These indexes are the RavenDB runtime objects that we spoke about previously.</p><p>Under the “List of Indexes” page, you can see which indexes exist and create your indexes. You can see if the index is healthy and customize it for your needs or even create yours.</p><p>For our demo, let’s take a quick look at the query that generates the “Product/Rating” index under the “Products” group. To save you the work, I got you covered and below is what you are supposed to find:</p><figure><img alt="RQL code for Index creation" src="https://cdn-images-1.medium.com/max/1024/1*9pts0QpWQGka3Sn87IbpSw.png" /><figcaption>RQL code for Index creation</figcaption></figure><p>We can explode your nested data structures in a breeze in real-time, which enables you to question your data without the need for unnecessary development sprints.</p><p>Thanks to <a href="https://www.cs.utah.edu/~germain/PPS/Topics/structures.html">dot notation</a>, creating ad-hoc components for your specific requirements or upgrading these into reusable <a href="https://www.techopedia.com/definition/24771/modular">modules</a>. Thanks to it, it is possible to reuse existing business logic from your data pipelines by any number of databases.</p><figure><img alt="Query Results using RQL" src="https://cdn-images-1.medium.com/max/1024/1*guTueEKb99N8kfygFaXESQ.png" /><figcaption>Query Results using RQL</figcaption></figure><h3>Yeah, it seems ok I guess; let’s wrap up while I share some final considerations.</h3><p>You’re still there, that’s great! With this publication, I hope it became clear what RavenDB is. But it is always good to remember where we should use it and how. While it is an excellent high-scalable option, it is good to remember that there’s no <a href="https://danielpaes.ca/nosql-for-data-storage-11e054ba0746">silver bullet</a> when it comes to data storage.</p><p>The use case of RavenDB is for faster data integration with Multidimensional databases in mind. With that said, RavenDB provides your a quicker response time when interacting with your data as it acts on the OLTP layer.</p><p>RavenDB capabilities are far from being all listed as it enables full-text queries, graph query capabilities and more accessible data migration. It is a complete data platform, and for more details, I would reinforce to visit this <a href="https://ravendb.net/features/#migration">link</a>; it could put your mind at ease if you think this might not be for you. If you are convinced and ready to rumble, take a look at their <a href="https://ravendb.net/learn/inside-ravendb-book/reader/4.0/1-welcome-to-ravendb">docs</a> to have the best of what it can offer.</p><p>You can even benefit from enhanced data lake capabilities. And some might want to explore more how to correlate RavenDB with Object storage options such as <a href="https://min.io/">MinIO</a> or <a href="https://ceph.io/en/">Ceph</a>. You will have a blasting performance when loading your OLAP layer using either their OLAP version or aligned with another tool such as <a href="https://www.getdbt.com/">DBT</a> for data ingestion or providing GitOps capabilities thanks to <a href="https://docs.lakefs.io/">LakeFS</a>.</p><p>See you guys next time!</p><p><em>Resources:</em></p><p>RavenDB — <a href="https://ravendb.net/">https://ravendb.net/</a></p><p>RavenDB Developer guide —<a href="https://ravendb.net/learn/inside-ravendb-book/reader/4.0/1-welcome-to-ravendb">https://ravendb.net/learn/inside-ravendb-book/reader/4.0/1-welcome-to-ravendb</a></p><p>RavenDB Studio — <a href="http://live-test.ravendb.net/studio/index.html#dashboard">http://live-test.ravendb.net/studio/index.html#dashboard</a></p><p>MinIO — <a href="https://min.io/">https://min.io/</a></p><p>Ceph — <a href="https://ceph.io/en/">https://ceph.io/en/</a></p><p>DBT — <a href="https://www.getdbt.com/">https://www.getdbt.com/</a></p><p>LakeFS — <a href="https://docs.lakefs.io/">https://docs.lakefs.io/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8758b8c0d3cc" width="1" height="1" alt=""><hr><p><a href="https://medium.com/codex/leveraging-ravendb-as-nosql-option-for-document-storage-8758b8c0d3cc">Leveraging RavenDB as NoSql option for Document storage</a> was originally published in <a href="https://medium.com/codex">CodeX</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Hardening your public container registry images with quay.io, an alternative for Docker Hub]]></title>
            <link>https://medium.com/coderbyte/hardening-your-public-container-registry-images-with-quay-io-an-alternative-for-docker-hub-79ef6fdb8040?source=rss-4e465c109e49------2</link>
            <guid isPermaLink="false">https://medium.com/p/79ef6fdb8040</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[Daniel Paes]]></dc:creator>
            <pubDate>Fri, 07 May 2021 13:20:45 GMT</pubDate>
            <atom:updated>2021-05-07T13:20:45.354Z</atom:updated>
            <content:encoded><![CDATA[<h4>When local Container Registry is not an option, deploy images on a public registry with fewer worries about data leaks.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Ac7yegXmi5_izy0V" /><figcaption>Photo by <a href="https://unsplash.com/@etiennegirardet?utm_source=medium&amp;utm_medium=referral">Etienne Girardet</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>How’s everything going? I hope all good!</p><p>Today I want to go back on vulnerability scans, a topic that seems to be usually forgotten by newcomers on the development of microservices for data analytics pipelines. Nothing wrong with focusing on smaller images; I also believe in having smaller images is better.</p><p>However, slimmer images and secure environments are not synonyms (unfortunately). And here, I would like to present quay.io, a Red-hat public Container Registry Image Repository with Clair under the hood. Running automated vulnerability scans on pushed Container Images. It becomes a good alternative for cases when creating a secured focused local Container Registry is not a viable option.</p><p>We will see why choosing quay.io for small projects instead of Docker Hub could be a good idea — ending up pushing a local Container Image into quay.io with the help of podman. So let’s get going.</p><h3>What does Docker Hub do?</h3><p>Docker Hub is a public container registry; it is a remote container registry for containerized applications we build locally. This tutorial will use podman due to its native support for rootless containers. However, you can replace podman for docker to replicate the commands if docker is your preferred container runtime tool — more information on how to run docker rootless on this <a href="https://docs.docker.com/engine/security/rootless/">link</a>.</p><p>Let us pick Python as an example. Below we can see the landing page for tags management at Docker Hub. It has some helpful information like the Operational System and the command to pull the desired tag locally.</p><p>Repository tag details on Docker Hub for Python</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/920/1*6LZ19zHt7VNkzgmWGnCHWw.png" /><figcaption>Repository tag details on Docker Hub for Python</figcaption></figure><h3>What does Quay provide?</h3><p>We can see some interesting extra information such as “security scan” and “manifest” on the similar landing page for the repository tags, as shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dqpvZVLhlCHnNNWZkPNaMQ.png" /><figcaption>Repository tag details for Python on Quay.io Repository</figcaption></figure><p>As Quay.io uses Clair to scan their images, we will narrow our analysis to only this tool. Clair is an open-source project managed by MITRE Corporation focused on scanning vulnerabilities based on the “Common Vulnerabilities Exposures” database (CVE, for short). Go to this <a href="https://www.redhat.com/en/topics/containers/what-is-clair">page</a> for more detailed information on vulnerabilities stored in this database. I would also invite you to check their official website on this <a href="https://cve.mitre.org/index.html">link</a>.</p><p>As mentioned, we can have the entries of CVE found vulnerabilities on the container, providing its CVE database detailed information such as package found and severity score, listing some.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GL4eF34vglpVmAnMs0hC-Q.png" /><figcaption>Container scan performed on quay.io for tag &quot;python:3.8.9-Debian-10-r25.&quot;</figcaption></figure><p>Further analysis of packages used by the image is possible without any extra configuration, as shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*P3A6LpTiniBNc0lhm30F8A.png" /><figcaption>Package scan on container tag published on quay.io for title &quot;python:3.8.9-Debian-10-r25.&quot;</figcaption></figure><h3>How to deploy a Streamlit image on quay.io</h3><p>For our tutorial, we will upload a containerized Streamlit application. This application provides an interactive dashboard upon which you can test your Machine Learning models. For more information, please discover more about it at their official <a href="https://streamlit.io/">homepage</a>.</p><p>The usual steps are to first log into your quay.io account (previously created account used to login into quay.io).</p><p>It is then needed to inform the id of a running container or one that successfully ran your application as this deployment is of a specific version of our application. The return of command “podman ps -a” will provide all containers running or running on our environment.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/965/1*9GeZJk8Xo2GqCujPUtkihw.png" /><figcaption>Local versions of containerized application</figcaption></figure><p>Once we define the desired id, we then commit our image locally to push it into our quay.io destination repository. A good practice to be adopted here is to have a base image more generalist and different versions better suited for your end-users needs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/977/1*U6kIhy0V3DbRwhlYCQIF3w.png" /><figcaption>Commit of desired containerized version into the Quay.io Container Registry</figcaption></figure><p>Below are the security scan results for our application on our newly deployed version found by quay.io scans with no extra configuration.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-UJ2JwtKV9noGz3zZmvYow.png" /><figcaption>Vulnerability scans report from a containerized application.</figcaption></figure><h3>Conclusions</h3><p>Here we saw quay.io as an alternative for those more concerned about the security scans. Due to its native implementation of Clair, it becomes easier to deploy your containerized applications while running vulnerability checks without any extra configuration. Also, the functionalities showed here could be locally deployed as an additional service on your Kubernetes pod, for example.</p><p>Doing so, extra tools like Cilium, Calico could add additional layers of security checks, and some could even extend Clair’s capabilities enabling the management of user-defined policies with Anchore. It is good to point out that Quay.io has Docker Hub distributions as the source for some images available on quay.io, with extra security scans in place. A good example is the “minideb” image found on this <a href="https://quay.io/repository/bitnami/minideb">link</a>. Making quay.io an option to Docker Hub when security checks on containerized applications are challenging to check.</p><p>See you guys next time.</p><p><em>Resources:</em></p><p>What is Container Registry — <a href="https://searchcloudcomputing.techtarget.com/definition/container-registry">https://searchcloudcomputing.techtarget.com/definition/container-registry</a></p><p>Should you use the — user flag in rootless containers? — <a href="https://www.redhat.com/sysadmin/user-flag-rootless-containers">https://www.redhat.com/sysadmin/user-flag-rootless-containers</a></p><p>What is a CVE? — <a href="https://www.redhat.com/en/topics/security/what-is-cve">https://www.redhat.com/en/topics/security/what-is-cve</a></p><p>How to push images into quay.io — <a href="https://quay.io/tutorial/">https://quay.io/tutorial/</a></p><p>Mitre Corporation — <a href="https://www.mitre.org/">https://www.mitre.org/</a></p><p>Podman Documentation — <a href="https://podman.io/">https://podman.io/</a></p><p>What is Cillium — <a href="https://docs.cilium.io/en/v1.10.0-rc1/intro/#intro">https://docs.cilium.io/en/v1.10.0-rc1/intro/#intro</a></p><p>What is Callico — <a href="https://docs.projectcalico.org/about/about-calico">https://docs.projectcalico.org/about/about-calico</a></p><p>Anchore Open Source Main Page — <a href="https://anchore.com/opensource/">https://anchore.com/opensource/</a></p><p>Quay.io minideb image — <a href="https://quay.io/repository/bitnami/minideb">https://quay.io/repository/bitnami/minideb</a></p><p>Making some sense from microservices and monolithic architectures — brief storytelling — <a href="https://danielpaes.ca/making-some-sense-from-microservices-and-monolithic-architectures-a-brief-storytelling-1caf2c16aa49?sk=75c8381a5234d9519299187b14832c04">https://danielpaes.ca/making-some-sense-from-microservices-and-monolithic-architectures-a-brief-storytelling-1caf2c16aa49</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=79ef6fdb8040" width="1" height="1" alt=""><hr><p><a href="https://medium.com/coderbyte/hardening-your-public-container-registry-images-with-quay-io-an-alternative-for-docker-hub-79ef6fdb8040">Hardening your public container registry images with quay.io, an alternative for Docker Hub</a> was originally published in <a href="https://medium.com/coderbyte">Tech x Talent</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Hopping into microservices]]></title>
            <link>https://medium.com/keyrus-canada/hopping-into-microservices-aa72a0ad62ab?source=rss-4e465c109e49------2</link>
            <guid isPermaLink="false">https://medium.com/p/aa72a0ad62ab</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[development]]></category>
            <dc:creator><![CDATA[Daniel Paes]]></dc:creator>
            <pubDate>Mon, 03 May 2021 14:16:44 GMT</pubDate>
            <atom:updated>2021-05-03T14:16:44.186Z</atom:updated>
            <content:encoded><![CDATA[<h4>What is it, and what is on for you when going under</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*0d-mJ7gthGj67x_q" /><figcaption>Photo by <a href="https://unsplash.com/@maxoor?utm_source=medium&amp;utm_medium=referral">Maxime Rossignol</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>How’s everyone going? Today I would like to go under microservice, things to have in mind when choosing a good strategy while migrating your on-premises systems (popular known as data centers) into cloud environments. I would present the concepts of moving it into cloud systems (lift and shift) or refactoring everything for optimized performance in cloud environments, ending with an example where it could fit your data pipelines and some things to have in mind when implementing it. So let’s get started.</p><h3>Getting our bases first, what is Microservices</h3><p>Microservices is a name that commonly comes up when discussing cloud migration projects, followed up by the point marks: “Should we lift and shift our environment? Can we afford to refactor it to save some big bucks on infrastructure costs? Even better, what if we can do a hybrid, so we will add new functionalities into our monolith deployments?”</p><p>Microservices are applications composed of smaller modular components, and these components work independently to fulfill a specific task. This approach brings up the possibility of detaching particular parts of your application making more straightforward enhancements on existing features or speeds up de deployment of new ones.</p><p>Docker and Kubernetes are commonly used to deploy Microservices. Docker handles tasks like containers registry and runtime execution while its orchestration is managed by Kubernetes, as demonstrated below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/767/1*63cA6IM8D6q4vvzaINAo3g.png" /><figcaption>Web Server NGINX Microservice Architecture Implementation Example — Source: <a href="https://www.nginx.com/blog/introduction-to-microservices/">https://www.nginx.com/blog/introduction-to-microservices/</a></figcaption></figure><h3>What are lift and shift and refactoring, some info on cloud migration strategies</h3><p>When discussing with clients the possibility of implementation of microservices for cloud migration projects, it is not that difficult to pop the question:</p><blockquote>For our Proof of Concept (POC), should we refactor or lift and shift our existing applications?</blockquote><p>As the name suggests, refactoring is the re-architecture of the existing application focused on optimal resource utilization. This approach is preferable when it is acceptable to sacrifice deployment time over the quality of the components for your existing applications. Retailoring your application to fit your needs better can have some unseen values attached, such as revalidating if your code functionalities and the used dependencies are still valid on the existing code while doing it.</p><p>On the other hand, lift and shift will deploy your existing application using a managed service from your cloud provider. While it saves time and allows your client to experiment faster, these applications are usually legacy systems or not optimized even for your local deployment. You are risking exceeding the costs due to the necessity of extra services to accommodate your application better.</p><h3>Implementing it on your data pipelines, with the help of Airflow</h3><p>Now that some basics concepts are covered. I want to show how we can use it on your data pipelines. Tools commonly used by large companies such as Banks and Telecom companies are BMC Control-M and Autosys. However, this has been changing with the adoption of open-source tools like Google’s Airflow, LinkedIn’s Azkaban or Spotify’s Luigi. In our article, I want to present Airflow’s KubernetesPodOperator.</p><p>It is good to have it clear as Kubernetes orchestrate how the components of an application are correlated. Airflow, on the other hand, dictates the flow on which multiple applications. A good rule of thumb is, when your requirements are specific to a single application and its components, you’ll probably need Kubernetes. In contrast, when constraints related to data quality and dependencies between its sources are required, we use the latter, common approach when ingesting data into Analytics environments.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1007/1*VZ4huot4kp_aUHUEWlVHhQ.png" /><figcaption>The architecture of Airflow&#39;s Kubernetes Pod Operator — Source: <a href="https://kubernetes.io/blog/2018/06/28/airflow-on-kubernetes-part-1-a-different-kind-of-operator/">https://kubernetes.io/blog/2018/06/28/airflow-on-kubernetes-part-1-a-different-kind-of-operator/</a></figcaption></figure><p>As the image illustrates (Blue Dot 1), the airflow scheduler triggers a Kubernetes pod using an API request into the Kubernetes cluster. This cluster will start a Kubernetes pod, which will resolve all the necessary environment configurations to deploy the desired service (Blue Dot 2). The logs for the process can be gathered on the Airflow or locally on the Kubernetes cluster (Blue Dot 3). When choosing to store any metadata on your Kubernetes nodes, it is always good to remember that due to Kubernetes’ ephemeral nature. Just remember to have good retention policies for your persistent metadata.</p><h3>Wrapping up</h3><p>Microservices is an excellent way to have a robust infrastructure where it can be easier to have new experimentations on your data with less development time when done correctly. Here we saw cloud migration strategies to deploy your applications and an example using Airflow, a data orchestration pipeline to start multicluster jobs on Kubernetes pods. It is good to remember that the modular deployment adds extra layers of complexity during the tests. More detached and complex your services are, its acceptance criteria become more complex, with further additional validations.</p><p>See you guys next time</p><p><strong><em>Resources:</em></strong></p><p>Lift and shift definition from Technopedia — <a href="https://www.techopedia.com/definition/33906/lift-and-shift">https://www.techopedia.com/definition/33906/lift-and-shift</a></p><p>Refactoring definition from Technopedia — <a href="https://www.techopedia.com/definition/3865/refactoring">https://www.techopedia.com/definition/3865/refactoring</a></p><p>Airflow on Kubernetes — <a href="https://kubernetes.io/blog/2018/06/28/airflow-on-kubernetes-part-1-a-different-kind-of-operator/">https://kubernetes.io/blog/2018/06/28/airflow-on-kubernetes-part-1-a-different-kind-of-operator/</a></p><p>Azkaban Documentation — <a href="https://azkaban.readthedocs.io/en/latest/index.html">https://azkaban.readthedocs.io/en/latest/index.html</a></p><p>Luigi Documentation — <a href="https://luigi.readthedocs.io/en/stable/">https://luigi.readthedocs.io/en/stable/</a></p><p>What is a web server — <a href="https://www.techopedia.com/definition/4928/web-server">https://whatis.techtarget.com/definition/Web-server</a></p><p>Nginx Introduction to microservices — <a href="https://www.nginx.com/blog/introduction-to-microservices/">https://www.nginx.com/blog/introduction-to-microservices/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=aa72a0ad62ab" width="1" height="1" alt=""><hr><p><a href="https://medium.com/keyrus-canada/hopping-into-microservices-aa72a0ad62ab">Hopping into microservices</a> was originally published in <a href="https://medium.com/keyrus-canada">Keyrus Canada</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Making data exploration less complicated with Python’s Modin, Mashmallow and Graphviz.]]></title>
            <link>https://medium.com/coderbyte/making-data-exploration-less-complicated-with-pythons-modin-mashmallow-and-graphviz-5256ae15a0f2?source=rss-4e465c109e49------2</link>
            <guid isPermaLink="false">https://medium.com/p/5256ae15a0f2</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[automation]]></category>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[data-modeling]]></category>
            <dc:creator><![CDATA[Daniel Paes]]></dc:creator>
            <pubDate>Sun, 25 Apr 2021 21:44:41 GMT</pubDate>
            <atom:updated>2021-07-12T12:24:26.184Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*86F8G_NMameuzKNuGQThvQ.jpeg" /><figcaption>Image: <a href="https://unsplash.com/photos/1-29wyvvLJA">Source</a></figcaption></figure><p>Hey folks, how are you guys doing?</p><p>Today I want to go back on some practical concepts around data exploration tasks and create a graphic representation of your semistructured data into a tabular format, used on relational databases systems using some code lines. We will generate an Entity-Relationship Model based on these new objects once it&#39;s all done. So let’s get started.</p><h3>What is the, not new at all, Entity-Relationship Model?</h3><p>Entity-Relationship models are a graphical representation of the data entities and their attributes. This schema presents the physical relationships between the entities and their attributes, making it a valuable tool for those who work on database development on the data exploration tasks. Below you can find an example, and in this article, we will go on how to create this example with some lines of code.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/648/1*UAQMMjlp9P69wyxX-TXuVw.png" /><figcaption>Automated Entity-Relationship Model created with Modin, Graphviz and Marshmallow.</figcaption></figure><h3>Ingesting your semistructured data into tabular format thanks to Modin</h3><p>We will ingest data from a semi-structured format in our tutorial; the data are stored in JSON (JavaScript Object Notation) files. We will then use Modin to receive this data into a DataFrame and generate a model validation on Marshmallow. This model will help to reinforce the source data types, once this validation is done we will render them with Graphviz.</p><p>We will use Modin under the Ray framework to replace the well-known Pandas due to its optimized approach to process larger datasets. It has the same built-in methods as pandas for data cleansing tasks. This article will demonstrate the serialization of semistructured data into a tabular structure.</p><p>It is allowing the same in-memory analysis achieved by Pandas with less computational power needed to do so. Applying an action known as flattening the data, we will create the same tabular format found on Relational Database Management Systems (or RDBMS for short). One proposed implementation is showed below:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/24e2ed2df53550799d674204d829e3e6/href">https://medium.com/media/24e2ed2df53550799d674204d829e3e6/href</a></iframe><p>It will generate a “flatten” DataFrame object, and the nested fields will have their column names using the dot notation format as shown below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/691/1*Acnzog4CRwbYpWS4bxAWXw.png" /><figcaption>Schema definition created from Dataframe results</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JwkBuihvgm3_SdBu3xJoeg.png" /><figcaption>New DataFrame Object made from flattened JSON</figcaption></figure><h3>Validating your data while creating your logical model thanks to Mashmallow</h3><p>As we can see, there is some noise coming from this file; also, we cannot create our model if we cannot define our entities. There are some primary tasks to be made before starting our Entity-Relationship model; these tasks reinforce the consistency of our data model, and one good way to do so is validating the source data types.</p><p>There are tools like SQLAlchemy, which implements the ORM model. These tools can increase the computational power needed for some tasks, and we will not need an entire ORM system; Marshmallow is a good faster alternative solution for our models, as it is not an ORM tool.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/350b64dfa0d536f0e1a7800c492c7ebd/href">https://medium.com/media/350b64dfa0d536f0e1a7800c492c7ebd/href</a></iframe><p>Below we can see what happens if we try to insert a row that does not follow the same structure as the one used while creating the DataFrame.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EQ45eDVP1CZ9OkeqR2W7lA.png" /><figcaption>Error while loading the table due to missing fields</figcaption></figure><h3>Giving it some colours with HTML Graphviz</h3><p>The structure built by Modin and Marshmallow becomes the perfect fit due to its tabular format of columns and rows to generate an Entity-Relationship Model. However, to better show our model, we will create three new DataFrame objects with derived fields from this original structure for our example.</p><p>Graphviz is a general application used to create graphical objects. This publication will use its Python library to render the created DataFrames into a schema based on Graphviz’s HTML tables tag support. Thanks to Graphviz, we can render these Dataframe objects into a graphical representation; below, you can see the code to create the structures used to generate the Entity-Relationship Model presented at the beginning of this publication.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e6b5117181e775bbecf7014fca7ad755/href">https://medium.com/media/e6b5117181e775bbecf7014fca7ad755/href</a></iframe><p>As we can see, we do not need to understand that much HTML development to start to create some helpful support documents. This result could be easily presented on tools such as Jupyter Notebooks or easily shared in commonly used formats such as png or pdf.</p><h3>Where to go from here</h3><p>Here we saw in a couple of lines of code how to implement some data integrity checks on your semistructured data and its graphical representation with a faster alternative to tools that we use on our daily tasks. Small integrity checks like the ones presented can enhance the quality of your data pipelines, removing some noises on your data; generating more accurate results with some extra lines of code.</p><p>See you guys next time</p><p><em>For further reference:</em></p><p>Jupyter Notebook: <a href="https://pypi.org/project/notebook/">https://pypi.org/project/notebook/</a></p><p>Modin: <a href="https://pypi.org/project/modin/">https://pypi.org/project/modin/</a></p><p>Graphviz: <a href="https://graphviz.readthedocs.io/en/stable/examples.html">https://graphviz.readthedocs.io/en/stable/examples.html</a></p><p>Marshmallow: <a href="https://pypi.org/project/marshmallow/">https://pypi.org/project/marshmallow/</a></p><p>Marshmallow dataframe plugin : <a href="https://pypi.org/project/marshmallow-dataframe/">https://pypi.org/project/marshmallow-dataframe/</a></p><p>HTML Tables docs <a href="https://www.w3schools.com/html/html_tables.asp">https://www.w3schools.com/html/html_tables.asp</a></p><p>What is Object Relational Mapper <a href="https://www.techopedia.com/definition/24200/object-relational-mapping--orm">https://www.techopedia.com/definition/24200/object-relational-mapping--orm</a></p><p>SQLAlchemy docs <a href="https://pypi.org/project/SQLAlchemy/">https://pypi.org/project/SQLAlchemy/</a></p><p>What is an RDBMS <a href="https://www.techopedia.com/definition/1235/relational-database-management-system-rdbms">https://www.techopedia.com/definition/1235/relational-database-management-system-rdbms</a></p><p>Entity-Relationship Model <a href="https://www.techopedia.com/definition/7057/entity-relationship-model-er-model">https://www.techopedia.com/definition/7057/entity-relationship-model-er-model</a></p><p>Database Models <a href="https://en.wikipedia.org/wiki/Data_modeling">https://en.wikipedia.org/wiki/Data_modeling</a></p><p>Data warehouse Modeling concepts on Kimball Methodology <a href="https://www.kimballgroup.com/data-warehouse-business-intelligence-resources/kimball-techniques/technical-dw-bi-system-architecture/">https://www.kimballgroup.com/data-warehouse-business-intelligence-resources/kimball-techniques/technical-dw-bi-system-architecture/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5256ae15a0f2" width="1" height="1" alt=""><hr><p><a href="https://medium.com/coderbyte/making-data-exploration-less-complicated-with-pythons-modin-mashmallow-and-graphviz-5256ae15a0f2">Making data exploration less complicated with Python’s Modin, Mashmallow and Graphviz.</a> was originally published in <a href="https://medium.com/coderbyte">Tech x Talent</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Skipping Regular Expressions (RegEx) for some data cleansing tasks with Natural Language…]]></title>
            <link>https://medium.com/keyrus-canada/skipping-regular-expressions-regex-for-some-data-cleansing-tasks-with-natural-language-a5926b434fda?source=rss-4e465c109e49------2</link>
            <guid isPermaLink="false">https://medium.com/p/a5926b434fda</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[analytics]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[nlp]]></category>
            <dc:creator><![CDATA[Daniel Paes]]></dc:creator>
            <pubDate>Mon, 19 Apr 2021 03:54:43 GMT</pubDate>
            <atom:updated>2021-04-21T02:26:45.876Z</atom:updated>
            <content:encoded><![CDATA[<h3>Expanding some data cleansing tasks with Natural Language Processing on Python</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/543/1*T-KKcVTc5NWOQED2vvfz1g.png" /></figure><p>Hey folks, how are you guys doing? Today I want to present another way of cleansing and to get information from your free text values. To do so, I would like to go quickly on some subdomains of Natural Language Processing and how it can help us on simple data cleansing tasks on our data exploration tasks—giving some examples of a tool on which is worth knowing. It could save some time and lines of code, keeping the same results on some functionalities, which would need whole data pipelines to achieve the same results; it seems like magic, right?</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fgiphy.com%2Fembed%2FNmerZ36iBkmKk%2Ftwitter%2Fiframe&amp;display_name=Giphy&amp;url=https%3A%2F%2Fmedia.giphy.com%2Fmedia%2FNmerZ36iBkmKk%2Fgiphy.gif&amp;image=https%3A%2F%2Fi.giphy.com%2Fmedia%2FNmerZ36iBkmKk%2F200.gif&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=giphy" width="435" height="292" frameborder="0" scrolling="no"><a href="https://medium.com/media/eb40b0be34eb5ffa898a078c565082d0/href">https://medium.com/media/eb40b0be34eb5ffa898a078c565082d0/href</a></iframe><h3>NLP in a punchline</h3><p>The NLP stands for Nature Language Processing, a subdomain of Artificial Intelligence focused on enhancing the interactions between machines and humans (here is additional information on this <a href="http://link.medium.com/K22luoGEzfb">link</a>). Extracting the text from a scanned document on your cellphone, you use Optical Character Recognition or OCR. Or those funny kid games with the google translator, also known as text-to-speech, both areas covered under the NLP umbrella. If you guys are interested in implementing these, take a look at <a href="https://pypi.org/project/pytesseract/">pytesseract</a> and <a href="https://pypi.org/project/text-to-speech/">text-to-speech</a>.</p><h3>Let use it for some cases with textblob</h3><p>For our examples, we will use textblob, which is a Python library used to handle textual data (aka free-text data). It extends the string function, so it can also be used to extract data from JSON structures for example, and it is also one good choice for tasks such as Part of Speech tagging and data tokenization to your Machine Learning models. Its utilization is mainly used for data cleansing in this area. However, I would like to present some excellent functions for some data ingestion pipelines and data exploration tasks.</p><p>A good part of our work on data cleansing is to make sense of some insufficient input data coming from different reasons, such as bad writing. Tools like textblob remove some of these tedious task with some lines of code.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/535/1*x7tIafx8pTeyEtDhqwPUqw.png" /><figcaption>The results of the correct command</figcaption></figure><p>Or in cases where you have a text from a column on which you need to translate into a different language (let&#39;s say English into French), you can use the translate function: or sometimes the source language is not informed, no problem at all! Textblob also allows discovering the source text&#39;s original language with detect_language (both use the Google API needing an internet connection to work, pay attention to those open ports on your ETL pipelines when using it).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/535/1*aBDpGqcJF9oqK5pNKUD3Xw.png" /><figcaption>The results of the corrected text from English into Spanish</figcaption></figure><h3>Final thoughts, can I go and implement it in my environment?</h3><p>It makes our life easier for most of our repetitive tasks when handling Text/String/Varchar datatypes. I could better fit on the exploratory phase of the data for most existing systems (&quot;What is this that I am analyzing/ingesting&quot;)?</p><p>While I feel it could save lots of time on the maintenance and support (Debugging this code could save some time to not go that deep on loop conditions during a weekend support call, in my opinion). Having it deployed could have extra constraints such as new dependencies on servers or updated artifacts to more recent image registries (impacting the current flow if not done correctly with things like Canary or Blue-green deployments).</p><p>While it is approved for new data pipelines won’t be a problem. It would be necessary some information about it to the team. So, in the end, there are lots of &quot;what if’s&quot; to say that the use of it on all your data pipelines is better. However, once managed, a power business user could enjoy robust and easy to maintain insights from free text with a more helpful toolkit in addition to the already powerful Regular Expressions (or RegEx).</p><p>See you guys next time!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a5926b434fda" width="1" height="1" alt=""><hr><p><a href="https://medium.com/keyrus-canada/skipping-regular-expressions-regex-for-some-data-cleansing-tasks-with-natural-language-a5926b434fda">Skipping Regular Expressions (RegEx) for some data cleansing tasks with Natural Language…</a> was originally published in <a href="https://medium.com/keyrus-canada">Keyrus Canada</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>