<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by German Osin on Medium]]></title>
        <description><![CDATA[Stories by German Osin on Medium]]></description>
        <link>https://medium.com/@gosin?source=rss-6ed1e3b217d8------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*zyFFNz7dQfl25nlpCerA-A.jpeg</url>
            <title>Stories by German Osin on Medium</title>
            <link>https://medium.com/@gosin?source=rss-6ed1e3b217d8------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 13:54:08 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@gosin/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Data compliance for Data Scientists]]></title>
            <link>https://medium.com/opendatadiscovery/data-compliance-for-data-scientists-a4e9b9717dce?source=rss-6ed1e3b217d8------2</link>
            <guid isPermaLink="false">https://medium.com/p/a4e9b9717dce</guid>
            <category><![CDATA[data-compliance]]></category>
            <category><![CDATA[data-observability]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-catalog]]></category>
            <category><![CDATA[data-discovery]]></category>
            <dc:creator><![CDATA[German Osin]]></dc:creator>
            <pubDate>Thu, 08 Dec 2022 12:54:37 GMT</pubDate>
            <atom:updated>2022-12-08T12:54:37.200Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9mMQ7EJbMItvTLZQT8FsHA.jpeg" /><figcaption>Image by freepik from <a href="https://www.freepik.com/free-vector/flat-design-data-privacy-facebook-cover_29068342.htm#query=data%20complaince&amp;position=7&amp;from_view=search&amp;track=ais">freepik</a></figcaption></figure><p>As a Data Scientist, you may often find yourself working with confidential and personal data. When developing models for behavioral segmentation or other purposes, it is important to be aware of the compliance implications of using this data.</p><p>There are several things to consider when determining whether or not to anonymize data:</p><p>1. The purpose of the model. If the model will be used internally only, there may be less of a need to anonymize data.</p><p>2. The sensitivity of the data. Some data is more sensitive than others and should be treated accordingly.</p><p>3. The laws and regulations surrounding data privacy. There are many laws and regulations governing how data can be used, collected, and stored. Be sure to familiarize yourself with these before using any confidential or personal data.</p><p>When in doubt, it is always best to err on the side of caution and anonymize data before using it for modeling purposes. This will help protect the confidentiality of your data and ensure that you are in compliance with all applicable laws and regulations.</p><p>The OpenDataDiscovery Platform provides a PII-sensitive search mechanism to assist in identifying confidential data using tags, labels, and metadata and, therefore, preventing potential monetary, legal, or reputational losses.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Z4D2IfzMZM5HHJMfGUxWUA.png" /></figure><p>The first step of this process is to go onto the ODD platform and search for tables. Check tags, labels as well metadata in order to find out which objects cannot be stored because they contain personal information like the customer&#39;s full name or age that should not appear publicly (PII). Anonymizing these would satisfy GDPR laws while still meeting PCI DSS standards so long we keep it down-low!</p><p>Despite popular belief, the data compliance process doesn’t have to be strenuous. By implementing software like <a href="https://opendatadiscovery.org/">opendatadiscovery</a>, you can manage and understand your data while keeping compliance in check. Not to mention, <a href="https://opendatadiscovery.org/">opendatadiscovery</a> is an open-source product so you can use it for free. Give it a try today and see how easy managing your data can be!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a4e9b9717dce" width="1" height="1" alt=""><hr><p><a href="https://medium.com/opendatadiscovery/data-compliance-for-data-scientists-a4e9b9717dce">Data compliance for Data Scientists</a> was originally published in <a href="https://medium.com/opendatadiscovery">Open Data Discovery</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Streamlining Data Quality Monitoring with Automation]]></title>
            <link>https://medium.com/opendatadiscovery/streamlining-data-quality-monitoring-with-automation-3781d9b8abac?source=rss-6ed1e3b217d8------2</link>
            <guid isPermaLink="false">https://medium.com/p/3781d9b8abac</guid>
            <category><![CDATA[data-quality]]></category>
            <category><![CDATA[data-monitoring]]></category>
            <category><![CDATA[data-discovery]]></category>
            <category><![CDATA[data-observability]]></category>
            <category><![CDATA[data-catalog]]></category>
            <dc:creator><![CDATA[German Osin]]></dc:creator>
            <pubDate>Tue, 01 Nov 2022 09:44:28 GMT</pubDate>
            <atom:updated>2022-11-01T09:44:28.724Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ie6AGxUs5JeRGQVHT-UHAw.jpeg" /><figcaption>Image by rawpixel from <a href="https://www.freepik.com/free-vector/illustration-quality-control_2806721.htm#query=data%20quality&amp;position=0&amp;from_view=search&amp;track=sph">freepik</a></figcaption></figure><p>Data quality is essential for any business. Without accurate data, it can be difficult to make sound decisions and achieve success. As a Quality Assurance Engineer, I know that it is impossible to manually monitor all data quality activities. That’s why I want to automate the process and introduce a system where we can track pipeline health on a given day. This will help us ensure that our data is always accurate and reliable.</p><p>Data tests are an important part of data quality management. By automating these tests, we can reduce the risk of human error and ensure that our data is always accurate. Additionally, by tracking pipeline health, we can quickly identify any issues that may arise. This will help us maintain a high level of data quality and avoid any potential problems.</p><p>Overall, automating data quality monitoring can help us improve the accuracy of our data and avoid potential problems. This will ultimately lead to better decision-making and improved success for our business.</p><p>We have a lot of data flowing through our system daily. For example, my pipeline is processing more than two billion orders and uses ten different dimensional tables as its sources!</p><p>I want to know how well my data is clean, accurate, and reliable. To check this I am going over six quality dimensions:</p><ul><li><strong>Timeliness</strong> — does it take too long for an order form meeples to become available in the product?</li><li><strong>Completeness</strong> — do I have any missing values or duplicated orders within the dataset</li><li><strong>Uniqueness</strong> — do I have any duplicated orders in my dataset</li><li><strong>Validity</strong> — do the values comply with the expected value format, e.g. book ISBN has an expected number of digits?</li><li><strong>Consistency</strong> — when I do a lookup at the dimensional table to return objects, does it cover all of them?</li><li><strong>Accuracy</strong> — will make sure of reconciliation with other sources</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Fms2FoO5_Uedq5WgWBLr5g.png" /></figure><p>The <a href="https://opendatadiscovery.org/">ODD Platform</a> is a one-stop shop for data quality solutions. It has integrated Great Expectations and Pandas, so you can use both at the same time! Plus users are able to import test suite results from any vendors as well</p><p>In order to measure the Timeliness, Completeness Uniqueness, and Validity of my project I decided to use Great Expectations.</p><p>As for the Consistency and Accuracy dimensions, I was looking for a tool that would allow me to compare many datasets, but I found this wasn’t possible in Great Expectations / Pandas. So instead, my team and I developed an SQL script that calculates KPIs by comparing certain values across all available datasets!</p><p>I’ve imported test suite results from Great Expectations to <a href="http://opendatadiscovery.org/">ODD</a>. In order to make sure that my custom framework test results are being picked up by the platform, I add them in as well directly from AirFlow.</p><p>As a result, I provided a transparent and accessible way of pipeline health monitoring and also use this feature when assessing the reliability of other sources of my interest.</p><p>When it comes to monitoring the health of my pipeline and assessing reliability, I turn to <a href="https://opendatadiscovery.org/">OpenDataDiscovery</a>. It’s a great tool that provides me with the transparency and accessibility I need to make informed decisions. If you’re looking for a way to improve your own data sources, I recommend giving OpenDataDiscovery a try.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3781d9b8abac" width="1" height="1" alt=""><hr><p><a href="https://medium.com/opendatadiscovery/streamlining-data-quality-monitoring-with-automation-3781d9b8abac">Streamlining Data Quality Monitoring with Automation</a> was originally published in <a href="https://medium.com/opendatadiscovery">Open Data Discovery</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Data Deprecation for Data Engineers]]></title>
            <link>https://medium.com/opendatadiscovery/data-deprecation-for-data-engineers-5e8d2471bbd3?source=rss-6ed1e3b217d8------2</link>
            <guid isPermaLink="false">https://medium.com/p/5e8d2471bbd3</guid>
            <category><![CDATA[data-observability]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[data-democratization]]></category>
            <category><![CDATA[depreciation]]></category>
            <category><![CDATA[data-discovery]]></category>
            <dc:creator><![CDATA[German Osin]]></dc:creator>
            <pubDate>Tue, 25 Oct 2022 09:32:07 GMT</pubDate>
            <atom:updated>2022-10-25T09:32:07.082Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*u52Q17upwOlPujCsR2Il_Q.jpeg" /><figcaption>Image by macrovector from <a href="https://freepik.com/free-vector/_21078562.htm#query=data%20deprecation&amp;position=5&amp;from_view=search&amp;track=ais">freepik</a></figcaption></figure><p>If you’re a data engineer or analyst, you know that data is constantly changing. Because of this, it’s important to have a process for deprecating old data so that your team can continue to work efficiently and effectively.</p><p>As a data engineer, I have been initiating the process of deprecating my datasets. One such dataset is an outdated book catalog that our team inherited from the legacy system’s codebase; it takes up too much resources on day-to-day basis and we want to get rid of this item as soon as possible without jeopardizing anything else in the pipeline stability because if something goes wrong with one part then the whole thing might come crashing down around us!</p><p>The use of modern data discovery solutions provides an end-to-end lineage diagram that helps to examine the possibilities and identify owners so there’s no need for any manual intervention during the deprecation process.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Fhy7qpS2XnXE9feq.png" /></figure><p>As a Data Engineer, my job is to explore lineage diagrams and make sure that the data doesn’t Make Its Way into downstream systems. If there are pipelines or dashboards using this information then I’ll need to update them so they don’t point back at our object when we remove it later on down line — also marking people who own objects if necessary for the deprecation process goes smoothly!</p><p>As a result, the deprecation process is well-managed and all users are notified in advance. Risks associated with this action have been mitigated to ensure that there’s minimal disruption for your company during the transition period.</p><p>The data discovery process doesn’t have to be difficult. With <a href="https://opendatadiscovery.org/">opendatadiscovery</a> platform, you can make the deprecation process simple and smooth. We provide an easy-to-use platform that helps you find and manage your data. Try <a href="https://opendatadiscovery.org/">opendatadiscovery</a> today and see how we can help you streamline your data discovery process.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5e8d2471bbd3" width="1" height="1" alt=""><hr><p><a href="https://medium.com/opendatadiscovery/data-deprecation-for-data-engineers-5e8d2471bbd3">Data Deprecation for Data Engineers</a> was originally published in <a href="https://medium.com/opendatadiscovery">Open Data Discovery</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Data Catalog Applications: Overview and Use Cases]]></title>
            <link>https://medium.com/opendatadiscovery/data-catalog-applications-overview-and-use-cases-38cfc4827984?source=rss-6ed1e3b217d8------2</link>
            <guid isPermaLink="false">https://medium.com/p/38cfc4827984</guid>
            <category><![CDATA[data-observability]]></category>
            <category><![CDATA[use-cases]]></category>
            <category><![CDATA[data-catalog]]></category>
            <category><![CDATA[data-discovery]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[German Osin]]></dc:creator>
            <pubDate>Wed, 01 Jun 2022 11:07:39 GMT</pubDate>
            <atom:updated>2022-06-01T12:12:35.208Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zQUNMraTY0RluABre03oqA.jpeg" /></figure><p>During my career in Big Data, I have worked with many teams and used almost all methodologies available. The teams have varied in their size and maturity level, and the architectures they used spanned from relational databases to more modern concepts like data mesh and data lake house. Project scope, goals, and vision also varied significantly. For example, we helped automate internal processes, empower sales through DS and self-service BI, and ensure security and compliance at all levels. While the challenges were different, we constantly faced the problem of <strong>efficient data governance</strong>.</p><p>Today, data catalogs are viewed as a driver of data democratization and better governance. There are various tools to handle data, derive valuable insights from it, and stop bad actors from taking advantage of data. As more effective tools enter the market, it becomes harder to choose the right one for your organization and enterprise. You really need to do lots of benchmarking, to learn about pros and cons, specific features, and technical details for every data catalog before moving forward towards the implementation and use. (I did just that in one of my recent articles on <a href="https://medium.com/@gosin/finding-the-right-data-catalog-solution-a265a4b3c0c3">open-source data catalog solutions</a>.)</p><p>Having implemented many data solutions, my colleagues and I collected a few real-world use cases that cover all the ins and outs of data catalogs. To make it easier for you, I decided to illustrate them by using a synthetic project — building a dataset for bookshop sales — as an example.</p><p>Imagine an abstract book shop with both an online presence and a brick-and-mortar store. It is a mid-sized retail company that employs a data team facing various challenges related to data management. Specifically, they find it hard to deal with sensitive data shared by customers, avoid irrelevant data, update data objects, prevent downstream system failures, and more.</p><p>Keeping this imaginary bookstore and its data team in mind, check out these short overviews of the primary use cases. We will explore each use case in more detail in future articles.</p><h4>Data Compliance for Data Scientists</h4><p>When you develop any machine learning model, you need to follow specific compliance requirements for data security (e.g. GDPR, CCPA). This use case is about patterns of using customers’ personal information.</p><p>When ML-engineers develop a model to customize communication processes for various customer segments, they deal with confidential and personal data, and they need to determine whether or not anonymization is required. The wrong answer may lead to potential monetary, legal or reputational losses. One approach to PII protection is a tagging system that helps to label specific data objects.</p><h4>Newcomer Onboarding</h4><p>Before a new data expert starts performing tasks, they have to research the domain, processes, tools and resources of the organization. To integrate a new team member successfully, a mentor, a knowledge base, and an information-oriented approach are all needed. Even when all the actors and elements are aligned, onboarding can demand a lot of time and effort.</p><p>What if your organization onboards a new employee who has to work with data? In this case, having a centralized system storing all information about data sources, data owners, metadata descriptions, and data lineage diagrams in one place will be extra helpful.</p><h4>Deprecation for Data Engineers and Analysts</h4><p>Managing a transparent deprecation process includes the following steps:</p><ol><li>Identify outdated objects</li><li>Predict the results of deprecation (e.g. downstream system failure)</li><li>Inform all stakeholders to assess the risks and impact</li></ol><p>This use case demonstrates how a data catalog can help to easily find users and owners of various data objects, and to explore lineage diagrams that keep track of pipelines, dashboards, and views that depend on the deprecated objects.</p><h4>Data Quality and Visibility for Data QA Engineers</h4><p>When <a href="https://provectus.com/data-quality-assurance/">Data QA</a> teams do not rely on automated data quality monitoring, it can affect data visibility and cause breaches in data quality. This use case describes how to solve this problem by using a test suite results’ import.<br>Let’s revisit the book shop example. A QA engineer knows that some book orders can be mapped to wrong descriptions, such as name, publisher, or genre, or even miss crucial fields associated with an order. The team’s pipeline is processing more than two billion book orders daily, so manual review seems impossible. <br>An appropriate solution is to find a Data Catalog that is compatible with predefined libraries or custom frameworks, to be able to easily share test suite results with the team and other stakeholders.</p><h4>Data Preparation for Visualization Engineers</h4><p>Data catalogs can help data visualization engineers to prepare complex data, and to investigate the data in detail before creating a dashboard.</p><p>The engineer’s challenge is to create a dashboard that tracks digital inventory at peak demand (Black Friday, Christmas or Valentine’s Day), and maintains stable performance if more than five users have to access the report simultaneously. To avoid wasting time on manual analysis of metadata and basic profiling, it is recommended to examine data sources by using a tagging system and a meta data storage.</p><p>These features can be useful for the following steps:</p><ul><li>Limit the history depth to one day only</li><li>Pre-calculate all KPIs to display in a view</li><li>Select only the fields needed for a dashboard</li></ul><h4>Service Provider and Pre-Sales</h4><p>In the pre-sales phase, project solutions may repeatedly change over a short period of time, and it requires specialists from various technical fields of expertise. Sometimes an unfavorable combination of these factors may lead to monetary and reputational losses caused by disappointed customer expectations or conflicts between an existent architectural landscape and best practices of the organization.</p><p>This use case addresses how to examine an existing architectural landscape and gather information about a toolset for better project scope planning.</p><h4>Customer Churn Analysis for Data Scientists</h4><p>A faulty notification system in your data streaming pipeline may cause a problem when your ML models receive irrelevant or obviously dirty data.</p><p>For example, imagine you have an algorithm for <a href="https://provectus.com/case-studies/user-churn-prediction-with-ai/">customer churn prediction</a> that triggers a sequence of re-engagement emails. Meanwhile, a new version of your streaming ETL pipeline is released and causes issues with some dependencies that affect a training process — the model receives incorrect data. Consequently, re-engagement emails are sent to the wrong users.</p><p>To solve this issue, it is recommended to use a data <a href="https://opendatadiscovery.org/">observability solution</a> with end-to-end lineage, integrated with an OpenTelemetry system.</p><h4>Recommendation Engine with Observable and Manageable Pipelines</h4><p>Building training models for a recommendation system is not easy. The process of model training involves many complicated steps, technical tools, and integrations. Even a subtle change, for instance, a new version of the Airflow ETL pipeline, can cause failures and incidents when the model starts to receive obsolete data.</p><p>This use case is about how alerts and end-to-end lineage help avoid asynchronous run schedules of a model and an Airflow pipeline.</p><h3>Conclusion</h3><p>Every company that deals with data and desires to monetize it needs to learn how to govern its data properly. Among data catalogs, there are plenty of effective tools for data governance. However, it is not enough to simply obtain a tool; it is also important to know how to apply it in practice. Use cases based on real challenges and problems can be used as beacons that navigate to solutions suitable for your business.</p><p>Looking to learn more? Subscribe to the <a href="https://blog.opendatadiscovery.org/">Open Data Discovery (ODD) Platform</a> blog, or follow me for more news and updates. Stay tuned!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=38cfc4827984" width="1" height="1" alt=""><hr><p><a href="https://medium.com/opendatadiscovery/data-catalog-applications-overview-and-use-cases-38cfc4827984">Data Catalog Applications: Overview and Use Cases</a> was originally published in <a href="https://medium.com/opendatadiscovery">Open Data Discovery</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Open Data Discovery Specification: A Universal Standard for Metadata Collection]]></title>
            <link>https://medium.com/opendatadiscovery/open-data-discovery-specification-a-universal-standard-for-metadata-collection-60425061112c?source=rss-6ed1e3b217d8------2</link>
            <guid isPermaLink="false">https://medium.com/p/60425061112c</guid>
            <category><![CDATA[database]]></category>
            <category><![CDATA[data-discovery]]></category>
            <category><![CDATA[data-catalog]]></category>
            <category><![CDATA[data-governance]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[German Osin]]></dc:creator>
            <pubDate>Wed, 04 May 2022 14:50:12 GMT</pubDate>
            <atom:updated>2022-05-06T11:18:03.545Z</atom:updated>
            <content:encoded><![CDATA[<p>Data is the lifeblood of AI. It is data that most significantly contributes to the quality of solutions powered by machine learning.</p><p>A critical asset for any enterprise, data can be extremely challenging to take advantage of since:</p><ol><li>The volume of data in different forms and formats is growing rapidly;</li><li>Existing data solutions are numerous, yet they do not entirely cover the needs of IT organizations and data teams;</li><li>Data pipelines and data infrastructures are becoming too complex to handle effectively</li></ol><p>Data has become too complicated, and it takes more and more time for professionals to collect and manage it efficiently and at scale.</p><p>This article provides an overview of Open Data Discovery Specification and explains why it is the solution for resolving the challenges of data collection and data management.</p><h3>Open Data Discovery Specification 101</h3><p><a href="https://github.com/opendatadiscovery/opendatadiscovery-specification">Open Data Discovery Specification (ODD Spec)</a> is an open-source, industry-wide metadata standard that enables engineers to collect and export metadata from cloud-native applications, infrastructures, and other data sources.</p><p>ODD Spec defines a schema for collecting metadata and specifies the semantics of data discovery. Data source-agnostic by design, it can be integrated with various data tools through endpoints to receive their metadata.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*9HXyy-szRhczpLZg" /><figcaption><em>Image by the author</em></figcaption></figure><p>ODD Spec allows for building data catalogs with features such as data federation, proper end-to-end lineage, data quality assurance, company-wide observability, and discoverable ML assets.</p><h3>The Problem of Data and Metadata Discovery</h3><p>As the amounts of structured, semi-structured, and unstructured data continue to grow, organizations face challenges that never existed before. Today, any professional who deals with data has to answer the following questions:</p><ul><li>As a ML Engineer or Data Analyst, how do I find data that fits my use case and is scattered across multiple systems and owners? How do I find responsible data owners if I find data issues or have questions about a source?</li><li>As a Data Analyst, how can I improve my data profiling processes and define sensitive data more efficiently?</li><li>As a Data Steward, how do I merge data from different data catalogs that use proprietary non-interchangeable metadata formats? How do I build federated systems if existing data catalogs do not support them?</li><li>As a Data Engineer, how do I make sure that my depreciation strategy will not break downstream data entities?</li><li>As a Data Steward, how do I expand the functionality of existing data catalogs beyond static data assets like tables and schemas?</li><li>As a Data Engineer or ML engineer, how do I tackle building and documenting ML models and data pipelines in locked-in data catalogs?</li></ul><p>All of these questions bring us back to data discovery. Not only does it take over 30% of all time spent by data teams, but it also creates the problem of access to data. For teams building AI solutions, data access is the major barrier to implementing AI &amp; ML at scale within their organizations.</p><p>Some open-source initiatives are already trying to address these challenges.</p><p>Open-source data catalogs like <a href="https://www.amundsen.io/">Amundsen</a>, <a href="https://datahubproject.io/">DataHub</a>, and <a href="https://marquezproject.github.io/marquez/">Marquez</a> are designed to reduce the time of data discovery. However, the data discovery process in these catalogs is pre-defined for the user and the possibility of re-using discovered metadata within other data discovery products is often very limited.</p><p>Marquez has introduced the <a href="https://github.com/OpenLineage/OpenLineage">OpenLineage</a> specification to standardize data lineage discovery. The specification does not cover critical entities that are not part of data lakes and warehouses: Dashboards, Pipelines, and ML Models. It does not allow for metadata enrichment through quality tests, test results, and data profiling.</p><p>Open Data Discovery Specification was developed to close the gaps that other tools and services have not yet covered, or do not cover entirely. The idea behind ODD Spec is simple:</p><p><em>ODD Spec provides the means and capabilities to unify metadata formats, enabling a more efficient, transparent, and consistent exchange of metadata from various sources, between multiple parties who need to discover data.</em></p><h3>Innovating the Data Discovery Space with ODD Spec</h3><p>We believe that Data and ML Engineers should not have to spend 90% of their time to ensure that data is clean and reliable. Tasks such as fine-tuning, debugging, and maintaining data pipelines, as well as cataloging and curating datasets, should not create data silos that prevent engineers from working on ML models, business-critical analytics, etc.</p><p>An open-source, <a href="https://towardsdatascience.com/the-missing-piece-of-data-discovery-and-observability-platforms-open-standard-for-metadata-37dac2d0503">open-standard ecosystem for collecting and managing data</a> can dramatically reduce the costs of building and maintaining data products for enterprises of all sizes.</p><p>This also concerns metadata. It should be easy to query and access through standard open API. Any data source, from DWH and data pipelines to ML model repositories, should support open API, to be self discoverable in any tool and any cloud.</p><p>Building on this line of thought, we have designed Open Data Discovery Specification with such core features as:</p><ol><li>Standard Open Data Discovery API (ODD API) democratizing data through open and transparent exchange of metadata between different systems: data sources, data pipelines, ML models, already existing data catalogs and next-gen ones;</li><li>Flexible data discovery models that are designed to be scalable in rapid processes of data landscape evolution and capture data entities at the moment of their creation;</li><li>Federated ‘Catalog of Catalogs’ for data discovery to unlock organization-wide data discovery;</li><li>Open-Source reference implementation based on the ODD API specification;</li><li>Community-driven structure to achieve better compatibility with a wide variety of third-party integrations and data tools</li></ol><p>We envision the data discovery process before vs. after ODD Spec as such:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VNLbfCm59nFs9qfvtdYWzg.png" /><figcaption>Image by author</figcaption></figure><p>In the currently existing data discovery ecosystem, various data sources (e.g. feature stores, ETL tools, ML pipelines, data warehouses, data quality tools, etc.) and data catalogs have to exchange data directly. Open Data Discovery Spec offers a standardized ODD Adapter that enables data exchange in a unified format.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*FjJvdvwo4NkfaHWr" /><figcaption><em>Image by the author</em></figcaption></figure><p>As can be seen from the image above, the data discovery process based on ODD Spec involves pull, push, and federation strategies. In this ecosystem, any data source and data catalog can expose an ODD Adapter API or use adapter microservices for data discovery. A push strategy can be applied to combine data with already discovered data entities. Data catalogs based on ODD Spec operate only metadata and never consume real data by their design. This approach ensures data compliance across all levels of organization.</p><p>With this in mind, it is clear that ODD Spec has the potential to benefit various groups of users, from data catalogs to enterprises. Let’s take a closer look.</p><h4><strong>Benefits of ODD Spec</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Om6W4BhG3rZShHrU" /></figure><p>The major benefit for data catalogs and data assets is realized through better integration with other tools and services. ODD Spec enables them to provide better, more easily accessible products to their customers, resulting in wider adoption on the market.</p><p>In the case of enterprise clients, major perks come with faster, more efficient evaluation of data, rapid access to trusted data sources, and more convenient exchange of metadata between teams, which leads to faster time-to-market for their products — be it advanced analytics dashboards, ML models feeding from various verified data sources, or real-world AI/ML-powered solutions.</p><h3>Data Discovery Models in ODD Spec</h3><h4>Push and Pull</h4><p>Metadata discovery with ODD Spec is similar to the collection of metrics, logs and traces, which can be done by using <em>pull</em> or <em>push</em> models. Because every model has its own range of applications, ODD supports both models to cover major use cases.</p><p>Pulling metadata directly from its source is the most straightforward method of metadata collection. However, it can be challenging to develop and maintain a centralized fleet of domain-specific crawlers. When metadata is pulled from multiple sources with no standard guiding the process, multiple source-specific crawlers must be created for each adapter. This is a complex and ineffective solution. ODD can resolve this issue with ODD Adapter, a universal API for metadata collection and processing.</p><p>ODD Adapter is a lightweight service that is integrated into a data warehouse as a proxy layer for metadata collection in a standardized format. The adapter receives requests for data entities and returns those entities in response. The adapters are designed to be source-specific and to expose only the information that can be pulled from a particular data source.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*9PXPS_0rPJs5DD9P" /><figcaption>Image by author</figcaption></figure><p>The pull model is recommended when you have good latency on index updates and have an adapter already integrated into the system.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*F2zXEIFDrT6m8QRb" /><figcaption>Image by author</figcaption></figure><p>The Push model supports the process where individual metadata providers push information to the central repository via APIs. The model is preferable for use cases like Airflow job runs and quality check runs.</p><h4>Data Discovery Federation</h4><p>Open Data Discovery comes with a built-in data federation feature, allowing multiple databases to function as one. It also allows for data entries to be efficiently scraped from other ODD servers.</p><p>Open Data Discovery covers two major use cases for data federation:</p><ol><li>Building scalable data catalogs</li><li>Pulling data entities from one ODD server to another</li></ol><p>It also employs two types of data federation.</p><h4>#1 Hierarchical Federation</h4><p>Hierarchical federation allows ODD servers to scale to environments with dozens of data centers and millions of nodes. The topology of hierarchical federation resembles a tree where higher-level ODD servers collect data entities from a larger number of subordinated servers.</p><p>A hierarchical setup, as seen in the image below, may include many ODD servers per datacenter, each of them collecting data in high detail (instance-level drill-down), and a set of global ODD servers that collect and store data from these local servers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*evrqVOKGCezNxn2B" /></figure><h4>#2 Cross-Service Federation</h4><p>In the case of cross-service federation, an ODD server of one service is configured to scrape selected data from another service’s ODD server, to enable queries against both datasets within a single server.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*TrveZGqO1J5QcnMp" /></figure><h3>Conclusion</h3><p>Open Data Discovery Specification is an open-source, industry-wide standard for metadata, designed to establish rules for how metadata should be collected, processed, and managed in an automated manner.</p><p>This article provides a brief overview of ODD Spec, with inroads into the challenge, the solution, major features and benefits, and the specifics of the data discovery process.</p><p>To learn more about Open Data Discovery Specification and our vision for data model specification, please check out <a href="https://github.com/opendatadiscovery/opendatadiscovery-specification/blob/main/specification/specification.md">specification.md on GitHub</a>. There you will find the code and examples you can use to explore the spec in more detail.</p><p>If you have any feedback or suggestions, feel free to reach out to me in the comment’s section, or contact me directly on <a href="https://www.linkedin.com/in/german-osin-47a9339/">LinkedIn</a>. Thanks!</p><p><em>P.S. I have recently shared what I think about data discovery and observability for ML solutions in an </em><a href="https://thesequence.substack.com/p/osin"><em>interview for TheSequence</em></a><em>. Please, check it out to get a bigger picture about ODD Spec.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=60425061112c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/opendatadiscovery/open-data-discovery-specification-a-universal-standard-for-metadata-collection-60425061112c">Open Data Discovery Specification: A Universal Standard for Metadata Collection</a> was originally published in <a href="https://medium.com/opendatadiscovery">Open Data Discovery</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Finding the Right Data Catalog Solution]]></title>
            <link>https://medium.com/provectus/finding-the-right-data-catalog-solution-a265a4b3c0c3?source=rss-6ed1e3b217d8------2</link>
            <guid isPermaLink="false">https://medium.com/p/a265a4b3c0c3</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[comparison]]></category>
            <category><![CDATA[data-catalog]]></category>
            <category><![CDATA[data-observability]]></category>
            <dc:creator><![CDATA[German Osin]]></dc:creator>
            <pubDate>Wed, 06 Apr 2022 15:16:27 GMT</pubDate>
            <atom:updated>2022-04-06T15:16:27.171Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*P0VZR1Cd7J6Rgkxy.png" /><figcaption>Image by <a href="https://ru.freepik.com/alexeybrin">alexey-brin</a> from <a href="https://ru.freepik.com/premium-photo/3d-render-of-abstract-shape-that-chaotically-deformed-data-centralisation-concept_17405594.htm">freepic.com</a></figcaption></figure><p>When organizations begin to adopt artificial intelligence, machine learning, and big data analytics, they realize that none of these technologies offers a magical fix that can change everything about their business overnight. In fact, to start driving actual value, they require fundamental, long-term changes in the organization’s culture, technology setting, and operating model. One key component that can enable a successful transformation is a data catalog, a management service that organizes your data in one place and lets you tag it with metadata, so you can discover and govern it more efficiently.</p><p>As someone who has been helping businesses to design and build robust solutions for handling data, I am going to share my experience about some of the most common data catalogs, to help you choose the right data catalog solution for your organization.</p><p>To begin, let’s take a closer look at data catalogs and the challenges they help resolve.</p><h3>Data Catalogs and the Challenge of Data Discovery and Observability</h3><p>Today, every business owner understands the importance of data, and the valuable insights it provides about your company and your customers. However, with an explosion in the volume of data, finding the right data has become harder than ever before. You can easily collect data, but what about further steps like data processing, data analysis, and BI? And what about data access, data discovery, data observability, and data governance?</p><p>In truth, a significant number of tech-savvy organizations, not to mention early adopters of any data- and intelligence-driven initiatives, find it challenging to understand the data they collect, account for how the data is moved around, and explain how and for what purpose data is used by different business divisions.</p><p>Unfortunately, the correct handling of data poses many challenges. These include:</p><ul><li>Difficulty finding and accessing the right data</li><li>Ineffective data collection: data is dumped vs being organized</li><li>Lack of understanding of the importance of data across the organization</li><li>Inefficient data processing: data lacks structure and is hard to understand</li><li>Inefficiencies in assessing the quality and trustworthiness of data in data swamps</li><li>Lack of infrastructure to capture, enhance, and reuse data assets in the organization</li><li>Reliance on manual and ad-hoc data preparation efforts</li></ul><p><strong>In short:</strong> Data has become too complicated. It takes increasingly more time for professionals to collect, process, analyze, and manage data efficiently and at scale.</p><p>To make matters worse, existing data catalog solutions do not entirely cover the needs of IT organizations and businesses. They lack standardized data collection methods, are incompatible with different catalogs, provide limited data lineage, and rely on inefficient data quality and observability practices.</p><p>As a result, data scientists spend lots of time discovering available datasets and making sure they can be trusted. Data science and ML engineering teams allocate about 90% of their time to ensuring that data is clean and reliable, and to tuning, debugging, and maintaining data pipelines that power mission-critical analytical dashboards and ML models. Enterprises are forced to hire armies of data stuarts, just to catalogize and curate data sets.</p><p>The challenge of data discovery, reliability, observability, and governance is real. The leading data catalogs are doing their best to make a difference in this arena. Below are some data catalogs that I can recommend for use in data- and intelligence-driven organizations.</p><h3>Top Data Catalogs for Enterprises</h3><p>Let’s kick off with a comparison table of open-source data catalog solutions, and continue by looking into each of them in more detail. Bear in mind that I have explored almost 30 different data catalog vendors, and narrowed it down to those featured below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*LjNDBWbMN3kBgQFe.png" /><figcaption>Image by author</figcaption></figure><h4>Amundsen</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*IcnjeolDipDvmUnR" /><figcaption>Image by author</figcaption></figure><ul><li>Pros: open-source, good community, extensive documentation</li><li>Cons: moderate UI, lack of ML entities</li></ul><p><a href="https://www.amundsen.io/">Amundsen</a> is an open-source data discovery and metadata engine that is designed and built to meet various needs of analysts and data scientists on one hand, and data engineers and software engineers on the other. The team claims that their solution can make data engineers, data analysts, and data scientists 20% more productive.</p><p>Amundsen has some of the best documentation, and developers call it the “Google of data cataloging.” It ranks data based on usage patterns. The UI itself can be characterized as moderate, since it does not provide space and cost metrics. It does, however, provide column statistics and a data preview. It supports Apache Hive, Amazon Redshift, Apache Druid, RDBMS, Presto, Snowflake, and several others listed on their GitHub page.</p><p><strong>Note: </strong>Amundsen has an SaaS version called <a href="https://www.stemma.ai/">Stemma</a>. It is a fully managed data catalog on its own, and features lots of useful integrations for data warehouses, orchestration and transformation, business intelligence, collaboration, security, and HR systems.</p><h4>Datahub</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*nkPX2uE0SdWeaooW" /><figcaption>Image by author</figcaption></figure><ul><li>Pros: open-source, good UI, lots of useful features (e.g. glossary and domains)</li><li>Cons: hard to install, custom ingestion protocol</li></ul><p><a href="https://linkedin.github.io/datahub/">Datahub</a> is an open-source project built by the people at LinkedIn. It has push and pull features, and its UI is modern and easy to use. The main issue with this data catalog is that it requires a lot of infrastructure effort, making it difficult to implement for smaller teams. It does not provide column statistics or metrics, and one minor flaw is that it supports only limited sources including Apache Hive, Apache Kafka, and RDBMS.</p><p>Datahub has an excellent <a href="https://demo.datahubproject.io/">demo</a> that lets you try out the product at a smaller scale. It also has an SaaS version called <a href="https://metaphor.io/">Metaphor</a> — a modern, scalable, and cloud-first data catalog and discovery service.</p><h4>Open Data Discovery</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*fIro2HwkINvea6KA" /><figcaption>Image by author</figcaption></figure><ul><li>Pros: good UI, ingestion based on specification</li><li>Cons: some features still in development</li></ul><p><a href="https://github.com/opendatadiscovery/odd-platform">Open Data Discovery (ODD)</a> is an open-source data discovery and observability platform. It is designed to help data-driven enterprises democratize data by making it more discoverable, manageable, observable, reliable, and secure. Because it supports <a href="https://github.com/opendatadiscovery/opendatadiscovery-specification">open data standards</a>, ODD enables data teams to make more efficient data exchanges between various data tools.</p><p>The UI of the platform is pretty good, and its ingestion is based on specification. Bear in mind that the platform is a work in progress, so some of the features are still in development.</p><h4>Apache Atlas</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*-725jvMMQXIJpzTC" /><figcaption>Image by author</figcaption></figure><ul><li>Pros: open-source, great metadata model</li><li>Cons: complex UI, a bit outdated</li></ul><p><a href="https://atlas.apache.org/#/">Apache Atlas</a> is an open metadata management and governance ecosystem that enables enterprises to meet their compliance requirements within Apache Hadoop. By supporting many integrations for classifying and governing data, it allows organizations to build a scalable catalog featuring various data assets, and to provide collaboration capabilities around those data assets for data scientists, analysts, and engineers.</p><p>All of this makes Apache Atlas a convenient tool for handling metadata. The challenge is that its ecosystem is a bit outdated (e.g. historically, a legacy product) and comes with a complex UI for advanced users.</p><h4>Marquez</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*5W8ACkM54gwOlc6y" /><figcaption>Image by author</figcaption></figure><ul><li>Pros: open-source, specification-based ingestion protocol, minimal infrastructure requirements to run</li><li>Cons: moderate UI, only a couple of adapters</li></ul><p>Originally created by WeWork, <a href="https://marquezproject.github.io/marquez/">Marquez</a> is an open-source metadata service that enables organizations to collect, aggregate, and visualize a data ecosystem’s metadata. It allows for the simplification of flexible data lineage queries across all datasets. It supports push functionality and works with Amazon S3 applications.</p><p>Marquez comes with a basic UI and only a couple of adapters. But these disadvantages are compensated for by a convenient specification-based ingestion protocol. The service does not require advanced infrastructure to run.</p><p><strong>Note:</strong> <a href="https://datakin.com/">Datakin</a> is an SaaS version of Marquez. It supports Apache Hive, Amazon RDS, Teradata, Amazon Redshift, Amazon S3, and Cassandra.</p><h4>Magda</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*X5LhWgawDkeYNbj_" /><figcaption>Image by author</figcaption></figure><ul><li>Pros: open-source, good for geodata, in-depth documentation</li><li>Cons: lack of adapters for Big Data solutions</li></ul><p><a href="https://magda.io/">Magda</a> positions itself as a federated open-source data catalog for all your big data and small data (e.g. Excel and CSV files). One area where Magda excels is geospatial and location data. If your organization works with geolocations, Magda might be the best option for you. It supports push features using its UI, and aside from Amundsen, is one of the few that support a data preview. But bear in mind that lack of adapters for big data can be a limiting factor.</p><h4>CKAN</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*CNP7iH7z91S90UOt" /><figcaption>Image by author</figcaption></figure><ul><li>Pros: open-source, thorough documentation</li><li>Cons: lack of adapters for Big Data solutions</li></ul><p><a href="https://ckan.org/">CKAN</a> is an open-source data management system for data hubs and portals. It provides capabilities for cataloging, storing, and accessing datasets with an API, featuring tools for handling data, data catalog, visualization tools, etc. CKAN is known for its rich documentation and support. However, the tool lacks some adapters for big data.</p><h4>OpenMetadata</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*KdjjLAdtZCWzgWys" /><figcaption>Image by author</figcaption></figure><ul><li>Pros: open-source, good UI, fast release cycles</li><li>Cons: complex installation, custom ingestion protocol, lack of ML entities</li></ul><p><a href="https://open-metadata.org/">OpenMetadata</a> is an open standard for metadata that provides a foundation for an end-to-end metadata management solution. It includes all necessary components for data discovery and governance, data quality, and observability. In combination with its quality UI, these features ensure ease of use and fast release cycles.</p><p>There are some key challenges with OpenMetadata, however. For instance, it is quite hard to install. It supports a custom ingestion protocol. And most importantly, it does not include entities for machine learning, which can be a decisive factor for data- and AI-driven enterprises.</p><h4>Atlan</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*pCbgvDqsjR-E-SJ4" /><figcaption>Image by author</figcaption></figure><ul><li>Pros: decent UI, well-funded startup, integrated data quality</li><li>Cons: custom ingestion protocol (requires access to your data from Atlan cloud), lack of ML entities</li></ul><p><a href="https://atlan.com/">Atlan</a> describes itself as a modern data workspace that helps data teams collaborate and democratize their enterprise data, while automating repetitive tasks. Its platform consists of three major components: data catalog and discovery, data lineage and governance, and data exploration and integrations. One of the perks of Atlan is that it features an integrated component for data quality assurance. Clean and unbiased data is key to advanced analytics and AI/ML projects.</p><p>One aspect where Atlan lags behind is its custom ingestion protocol. You will need to access your data from Atlan cloud to start working with it. It also lacks some ML entities.</p><p>Atlan supports Presto, Deequ, Atlas, Airflow, and Hudi.</p><h4>Alation</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*kktVRgiNI6dXi3mr" /><figcaption>Image by author</figcaption></figure><ul><li>Pros: proper UI, lots of great features like glossary, policies, etc.</li><li>Cons: custom ingestion protocol, lack of ML entities</li></ul><p><a href="https://www.alation.com/">Alation</a> is a data catalog that prioritizes the human aspect of handling data. It supports many useful features for faster onboarding, query re-use, and automated curation. On top of that, its UI is simple, consistent, and intuitive. If you are going to use Alation, however, bear in mind that it has a custom ingestion protocol and lacks ML entities. This is critical for businesses that want to go beyond simple analytics and business intelligence, into the AI/ML realm.</p><h4>Ataccama</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*QrQBWBRKhp5NIakv" /><figcaption>Image by author</figcaption></figure><ul><li>Pros: high-quality UI, integrated data quality, SaaS</li><li>Cons: custom ingestion protocol, lack of ML entities</li></ul><p><a href="https://www.ataccama.com/">Ataccama</a> is an AI-powered enterprise platform for data and metadata that features components for data quality, master data management, and data integration. The UI of the platform is good, and it enables fast analytical teams, highly regulated governance teams, and technical data teams to easily handle their data assets. One great perk of Ataccama is that it was designed with a strong focus on data quality. This is crucial for agile, data-driven organizations.</p><p>The downside is that Ataccama locks users in, because it requires a custom ingestion protocol. Though the situation is changing, it also lacks certain entities for machine learning.</p><h4>Monte Carlo</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*FMoOc2moWsvVbIDU" /><figcaption>Image by author</figcaption></figure><ul><li>Pros: decent UI, data observability features</li><li>Cons: custom ingestion protocol, lack of ML entities, focus on data observability as opposed to data discovery</li></ul><p><a href="https://www.montecarlodata.com/">Monte Carlo</a> is a data platform that helps data teams resolve the problem of data downtime, enabling them to handle their dashboards more efficiently, train more accurate ML models faster, and drive analytics operations. The platform is extensively focused on machine learning-powered data observability, which gives data teams an advanced capability to look into data and identify potential issues at scale. This gives Monte Carlo a considerable advantage over its competitors in data observability.</p><p>Like many other proprietary data catalog solutions, Monte Carlo locks the users into its own data ingestion protocol. It also lacks certain ML entities, which can be a deal-breaker for many organizations today.</p><h3>Conclusion</h3><p>Data catalogs are essential components of digital, data, and AI transformations. They organize all your data in one place and give you the tools to handle data more efficiently and on a larger scale. Data discovery, data observability, data governance, and other data-related activities can be improved and streamlined by incorporating a data catalog in your workflow.</p><p>I have explored almost 30 different data catalog vendors, and we have selected several top picks that you can use to compare and contrast different options. Some of them are open-source, some are proprietary, so you can pick and choose to find a solution that works for your team, organization, or company.</p><p>If you have any questions or suggestions, kindly share your feedback in the comment section, or reach out to me directly!</p><p><em>DISCLAIMER</em><br>This article offers my perspective on data catalog solutions. I did my best to be objective, but the community is sure to have something to add. Open discussion is always welcome and appreciated.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a265a4b3c0c3" width="1" height="1" alt=""><hr><p><a href="https://medium.com/provectus/finding-the-right-data-catalog-solution-a265a4b3c0c3">Finding the Right Data Catalog Solution</a> was originally published in <a href="https://medium.com/provectus">Provectus</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Data Discovery for ML Engineers]]></title>
            <link>https://medium.com/opendatadiscovery/data-discovery-for-ml-engineers-d26197e99bd6?source=rss-6ed1e3b217d8------2</link>
            <guid isPermaLink="false">https://medium.com/p/d26197e99bd6</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-governance]]></category>
            <category><![CDATA[data-discovery]]></category>
            <category><![CDATA[data-catalog]]></category>
            <dc:creator><![CDATA[German Osin]]></dc:creator>
            <pubDate>Tue, 05 Apr 2022 19:14:14 GMT</pubDate>
            <atom:updated>2022-05-09T10:40:44.462Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*nrmsLLjhePpd8eUr.jpg" /></figure><p>Real-world production ML systems consist of two main components: data and code. Data is clearly the leader, and rapidly taking center stage. Data defines the quality of almost any ML-based product, more so than code or any other aspect.</p><p>In Feature Store as a Foundation for Machine Learning, we have discussed how feature stores are an integral part of the machine learning workflow. They improve the ROI of data engineering, reduce cost per model, and accelerate model-to-market by simplifying feature definition and extraction.</p><p>While feature stores are important, they handle just one type of data that is specific to the ML workflow. There are many more types of ML-specific data, and as an ML Engineer, you should have a convenient way to discover, validate, and monitor all of them.</p><p>In the same way that a feature store acts as a warehouse of features to benefit machine learning, data catalogs can help you access all types of data entities (including feature stores) that data and ML engineers need in order to tackle both code and data aspects of ML models.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/875/0*435w2nUrV2SsaYi6.png" /><figcaption>Feature stores act as ML-specific sources of data that ML Engineers can discover for use in their ML workflows</figcaption></figure><p>This article provides an overview of a type of data catalog that could combine the efforts of DE and DS teams, facilitate data-centric MLOps, improve TTM of ML models and remove the barriers to scaling ML/AI efforts of organizations — and explains why all of this matters for ML Engineers.</p><h3>Why ML Engineers Need a Data Catalog Designed for Machine Learning</h3><p>Even without taking into account the ML Engineer’s workflow, it can be challenging to make the most of data, primarily due to the following trends:</p><ol><li>The volume and complexity of data formats are skyrocketing</li><li>Data pipelines are becoming too complex to be handled effectively with existing tools</li><li>Modern requirements for data discovery are evolving faster than solutions to address them</li></ol><p>The ML model lifecycle also poses additional challenges:</p><ol><li>ML pipelines are increasingly complex</li><li>Siloed discovery of ML and data entities prevents scaling of ML operations</li><li>Tools for the discovery and observation of ML entities along with data have not yet evolved</li></ol><p>Data catalogs and observability tools that exist today have not yet been integrated with ML entities — at least, not on the architectural level. While factoring in machine learning has been considered, it is evident that:</p><ol><li>Data discovery tools do not include ML-related data entities</li><li>ML entities are absent from lineage and observability tools</li><li>When data in the ML pipelines goes wrong, stakeholders are kept in the dark about it</li><li>Data and ML engineers are unaware of what is happening in the other’s part of the pipeline, and that hinders their collaboration and synergy</li><li>The collaboration gap between DE and ML teams causes black box issues that are hard to detect, which often leads to loss of profits and damaged reputations</li></ol><p>Since the advent of machine learning, ML-related entities (such as ML pipelines, ML models, features stores, etc.) have become integral to the data landscape. Despite this, and the role of data as a key component that defines the quality of ML products, they are still absent in existing data discovery and observability solutions.</p><p>Until recently, data catalogs and observability platforms were not designed to include ML entities as first citizens. The overall picture looked like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/875/0*f4vwpRfLLHWggXM4.png" /><figcaption>The ML world is absent from the current data discovery/observability ecosystem.</figcaption></figure><p>In our latest article entitled <a href="https://towardsdatascience.com/the-missing-piece-of-data-discovery-and-observability-platforms-open-standard-for-metadata-37dac2d0503">The Missing Piece of Data Discovery and Observability Platforms: Open Standard for Metadata</a>, we take a closer look at the state of data catalogs and observability platforms. We also explain why they are struggling to keep pace with the rapidly evolving demands of professionals in the data and ML spheres.</p><h3>Consequences of Siloed Data and ML Discovery</h3><p>At the outset of a machine learning project, the first step immediately after understanding the business use case is data discovery — the process of searching, understanding, and evaluating data to satisfy your goals. Since the quality of products powered by machine learning depends so closely on the quality of data, it is important to find data that can be trusted to work with your model.</p><p>Data catalogs are a tool category designed to help with the discovery step. But the problem is that none of the existing data catalogs were designed with the needs of ML Engineers and the ML model lifecycle in mind. While they offer data search and evaluation capabilities, you cannot use them to discover all of the ML-related entities from platforms where you discover other types of data.</p><p>ML-specific dataflow includes a lot of entities that are part of the ML lifecycle: ML pipelines, model experiments, model metrics, model artifacts, ML production services, etc.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*zxykIvgu9oH5NtKc.png" /></figure><p>The unnatural gap between data discovery and other steps of the ML workflow creates inefficiencies in managing ML-related data operations.</p><p>In order to build an ML model that can generate maximum business value, it is crucial to have a whole picture of where data originated, how it has evolved, and its specific characteristics.</p><p>Machine learning entities that cannot be discovered can disrupt and slow down the entire process of working with data in the context of ML. It is impossible to manage what cannot be discovered and evaluated. As a result, data-driven organizations find it increasingly challenging to scale ML to its full potential. Disregarding ML in the data discovery workflow also strips companies of crucial opportunities to scale their ML and AI operations.</p><h3>Consequences of Siloed Efforts of DE and ML Teams</h3><p>The absence of ML entities in data discovery tools also means that the data and ML engineering portions of the data pipeline are siloed, keeping each team in the dark about what’s happening on the other side. This lack of observability can potentially cause unpredictable and hard-to-detect black box issues.</p><p>A classic example of a black box failure caused by lack of cross-department collaboration is undetected data tardiness. Imagine that you have a data and ML pipeline for a website’s recommendation model that two departments/teams are responsible for: Data Engineering and Data Science. The DE team uses Airflow to ETL and publishes data to a feature store, and the DS team supports a Kubeflow training pipeline that uses data from the feature store.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/875/0*czUcoKXhn9i5401i.png" /></figure><p>Without proper observability in place, it can be almost impossible to notice occasional changes in the algorithm.</p><p>For example, if the DE team deploys a new version of the Airflow ETL pipeline, scheduled to run daily at 12:00 am, while the ML training pipeline runs daily at 1:00 am, it would be difficult to notice that the Airflow pipeline execution time increased from 1 to 1.5 hours, and now terminates after the ML pipeline begins, lagging behind.</p><p>The model would end up constantly running on obsolete data from the previous day, without taking into account new data gathered throughout the day.</p><h3>What ML-Related Data Can Be Gathered and Discovered?</h3><p>Data has become extremely complicated, and modern tools for keeping track of ML-related entities in the same ways that data catalogs keep track of other types of data have not yet evolved. Such a catalog could help to plan and run ML training, experiments and tests, improve model-to-production time, and make for cleaner model development.</p><p>These catalogs can involve complex entities and relationships, which is why it’s important to be in control, and to be able to track those entities automatically.</p><p>As an ML engineer, you can use existing data catalogs to discover proper training data. But what about other artifacts that should be integral to your work process?</p><p>A simplified ML model lifecycle looks like this:</p><ol><li>Understanding the business case</li><li>Data discovery (finding the right data for model training)</li><li>Running experiments</li><li>Training the model</li><li>Finding the right data for model inferencing</li><li>Model inferencing (running the ML model on new data to get predictions)</li></ol><p><strong>ML experiments. </strong>When working on an ML model, you have to run a lot of experiments — procedures to test your hypothesis (questions about your model, like which model performs better). You run experiments to see how different parts of a model come together under different circumstances. For example, you might want to:</p><ul><li>Use different models and model hyperparameters</li><li>Use different training or evaluation data</li><li>Run different code (including small code changes that you want to test quickly)</li><li>Run the same code in a different environment</li></ul><p><strong>ML Models</strong> are machine learning algorithms that generate predictions from data. As an ML Engineer, you definitely want different versions of ML model artifacts to be able to search and discover through a data discovery solution.</p><p><strong>ML model metrics. </strong>Model evaluation metrics (like accuracy, precision, recall, specificity, log loss, and other scores) are used by ML Engineers to evaluate how their models perform. They are crucial for any data science project, and serve as a guiding light in helping to understand what direction to take next.</p><p><strong>Data for ML model inferencing.</strong> Inferencing is the process of using a trained algorithm on new data to drive predictions. Just as with training and testing data, you also need to discover data for the inferencing stage, and make sure it fits your requirements.</p><p><strong>Feature Stores. </strong>Feature store is a data warehouse that ingests feature data from different sources after it has been transformed, aggregated and validated. Data from feature stores can then be used for model training and inferencing stages.</p><h3>How Integrated ML Workflow, Data Discovery and Observability can Benefit Data Teams</h3><p>The typical workflow of an ML engineer goes through the following steps: search what data exists in an organization, evaluate that data against the project and define its quality, get proper access to the data, use it to run experiments, pick the best hypothesis, run ML model training on the data in Kubeflow, register the model with a model registry, and connect it to feature store to ensure proper inferencing.</p><p>When an ML engineer embarks on a project, it is important to operate high-quality, reliable data in each of these steps. Even when a model is ready and deployed, it completely depends on the underlying data. If data breaks, so does the model. Changes that may seem harmless at first glance, like slight shifts in distribution or minor data lateness, have the potential to quickly snowball into major issues.</p><h3>Open Data Discovery Platform for ML and Data Engineers</h3><p>In thinking about the above-mentioned issues, we developed a solution with ML and Data Engineers in mind. ODD Platform is the first open-source data discovery and observability platform. It fills the missing parts of data discovery to allow ML engineers to easily find all entities that are a part of their ML workflow. It also helps data engineers to discover everything they need for data engineering.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*P3ptYZMbgEioLHnm.png" /></figure><p>ODD automatically collects metadata from all connected data sources (including ML-specific ones) and registers them for the lineage. The lineage section of each data product shows all its upstream and downstream connections. If something changes in the data that ML engineers use for their ML models, lineage and rich alerts will help them proactively find the root cause of the issue in a matter of minutes, instead of hours or days. All owners of affected data products will be immediately notified of changes.</p><p>ODD Platform is based on Open Data Discovery Specification — an open-source, industry-wide standard for metadata, designed to establish rules for how metadata should be collected, processed, and managed in an automated manner.</p><p>Unlike existing data catalogs and other data observability tools, ODD does not require any paid or complex infrastructure. The deployment process is as easy as it gets. The only difficult requirement is having PostgreSQL. ODD runs as a cluster of microservices, and you can easily add or remove data sources from your setup.</p><p>ODD is licensed under the Apache 2 license. The team has committed to staying forever open-source, and to building the product in public without any hidden agenda.</p><h3>Conclusion</h3><p>Data quality is crucial for building sustainable and reliable ML models. To have relevant, reliable and readily available data at hand at all times, ML engineers need a data discovery and observability system that operates ML entities as first-class citizens, to provide company-wide data discovery, quality assurance, and transparency.</p><p>Such a tool would remove barriers that currently exist between data engineers and data scientists, and facilitate collaboration. It would end the disconnect that people in those roles currently experience, and make them fully aware of the other’s part of the data pipeline.</p><p>ODD Platform is the first open-source data discovery and observability platform for ML engineers that offers a solution to the set of issues discussed in this article. Following are the main use cases it can help with:</p><ol><li>Onboarding of new ML Engineers and other data professionals</li><li>Data discovery for ML engineers and other data professionals</li><li>Fully end-to-end data observability and alerting for data pipelines</li></ol><p>For more information about ODD Platform, please check out our organization<a href="https://github.com/opendatadiscovery"> opendatadiscovery</a> and the main repository<a href="https://github.com/opendatadiscovery/odd-platform"> odd-platform</a> on GitHub.</p><p><em>Originally published at </em><a href="https://www.datasciencecentral.com/data-discovery-for-ml-engineers/"><em>https://www.datasciencecentral.com</em></a><em> on April 5, 2022.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d26197e99bd6" width="1" height="1" alt=""><hr><p><a href="https://medium.com/opendatadiscovery/data-discovery-for-ml-engineers-d26197e99bd6">Data Discovery for ML Engineers</a> was originally published in <a href="https://medium.com/opendatadiscovery">Open Data Discovery</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[ German Osin/Provectus About Data Discovery and Observability in ML Solutions]]></title>
            <link>https://medium.com/opendatadiscovery/german-osin-provectus-about-data-discovery-and-observability-in-ml-solutions-8c3d9b2a28c3?source=rss-6ed1e3b217d8------2</link>
            <guid isPermaLink="false">https://medium.com/p/8c3d9b2a28c3</guid>
            <category><![CDATA[data-discovery]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[mlops]]></category>
            <category><![CDATA[data-catalog]]></category>
            <category><![CDATA[data-observability]]></category>
            <dc:creator><![CDATA[German Osin]]></dc:creator>
            <pubDate>Wed, 15 Sep 2021 12:13:09 GMT</pubDate>
            <atom:updated>2022-05-09T13:06:47.118Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*c2l-UL-_-rwPOtki.png" /></figure><p><em>It’s so inspiring to learn from practitioners. Getting to know the experience gained by researchers, engineers, and entrepreneurs doing real ML work is</em><strong><em> a great source of insight and inspiration</em></strong><em>. Share this interview if you find it enriching. No subscription is needed.</em></p><h3>👤 Quick bio / German Osin</h3><blockquote><strong><em>Tell us a bit about yourself. Your background, current role and how did you get started in machine learning?</em></strong></blockquote><p><strong>German Osin (GO):</strong> I am a Senior Data Platform Architect at <a href="https://provectus.com/">Provectus</a>. I lead several of the company’s open-source products and help build advanced data platform solutions. A big part of my job is bringing AI/ML to data, enabling critical data processing for analytics, deep insights, and Business Intelligence.</p><p>My first big experience with ML was in 2013 when I worked on a system for modeling look-alike audiences for targeted advertising. My prior experience was in Big Data and near real-time solutions. So initially, I approached the new challenge as a classical Big Data problem. But it made me take a deep dive into the subject and sent me directly into the ML world, helping me explore it more in-depth. I ended up using the full ML stack and am very happy about it.</p><h3>🛠 ML Work</h3><blockquote><strong><em>Provectus has a long history of helping enterprises implement ML solutions in the real world, and recently open-sourced a new data discovery and observability platform. Can you tell us a bit more about the inspiration for this new project, and elaborate on some of its technical capabilities?</em></strong></blockquote><p><strong>GO:</strong> We draw a lot of inspiration from our clients and their stories. After talking to them about the issues they experience when trying to implement ML, we discovered several patterns:</p><ol><li>Despite the advent of data catalogs, the majority of data teams (up to 90%) still complain about their data access and discovery experience, being unable to access or find data they need for work.</li><li>Data discovery still eats up too much time. About 25–50% of data specialists’ working time funnels into finding correct and trustworthy data.</li><li>Due to the rapid development of ML and Big Data industries, the bar has been set much higher for products that address these issues. They need to offer not only company-wide lineage and data quality assurance but also encompass the entire ML world. Otherwise, organizations are held back from applying ML/AI at scale.</li></ol><p>We explored the market and confirmed that there is nothing close to this type of product available, so we decided to design one ourselves.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*fkrLDUjEG3JQ7NT7.png" /></figure><blockquote><strong><em>Data discovery is an area where Provectus has been doing a lot of work recently. What are some of the best practices that companies should consider to enable data discovery controls in ML pipelines, and how do they differ from traditional enterprise data catalog approaches?</em></strong></blockquote><p><strong>GO:</strong> With the advent of the data-centric approach in MLOps, several things are becoming increasingly important:</p><ol><li><strong>Building data discovery products based on an open-source standard for metadata exchange.</strong> This allows for extensive interoperability and makes it easy to quickly plug in any new data sources, tools, and other catalogs when you need them in your system. Check out this <a href="https://towardsdatascience.com/the-missing-piece-of-data-discovery-and-observability-platforms-open-standard-for-metadata-37dac2d0503">research article about why building on an open metadata standard is so important</a>.</li><li><strong>Gathering as much metadata as possible to get the full picture.</strong> In the data discovery process, metadata helps us evaluate data objects and their trustability. The more metadata you have, the better your results. Modern data discovery platforms need to serve people working in many different roles and relying on all kinds of metadata for evaluation. A system that offers more metadata enhances their efficiency in finding data and doing their jobs.</li><li><strong>Avoiding manual metadata collection. </strong>Products that you implement should not require any manual work to retrieve the necessary data. Data retrieval should be a straightforward automated collection process that everyone in your company can use, and that does not require a lot of time or infrastructure resources.</li></ol><p>By contrast, traditional enterprise data catalogs lock up metadata and are not flexible enough to include all the tools a company might need. Sharing metadata is often not in a company’s best interest, so the catalogs were not designed to do so.</p><p>Additionally, traditional catalogs completely ignore the world of ML. They overlook ML training jobs, model repositories, model instances, and feature stores that are rapidly becoming a <a href="https://towardsdatascience.com/feature-store-as-a-foundation-for-machine-learning-d010fc6eb2f3">fundamental component of the ML stack</a>. The absence of ML from data lineage leaves a company exposed to unknown and unpredictable black-box issues that can lead to numerous downsides for the business.</p><p>We consider it crucial to include ML entities as a part of data discovery and lineage, get truly end-to-end observability, and be proactive about black-box issues. We have implemented these principles in the <a href="https://github.com/opendatadiscovery">ODD Platform</a> we’re currently working on.</p><p>Also, traditional data catalogs rely on manual metadata collection, which is obsolete in the modern world and will never pay off.</p><p>If you’d like to learn more about data discovery and our approach to resolving various data challenges, watch this webcast: <a href="https://provectus.com/data-discovery-and-observability-platforms/?utm_source=tsq&amp;utm_medium=interview&amp;utm_campaign=odd">First open-source data discovery and observability platform for ML Engineers</a>.</p><blockquote><strong><em>Data management methods in ML solutions can be vastly different types of datasets, such as tabular, text, audio or images. What are some of the commonalities and differences for enabling data observability and lineage analysis across different types of datasets?</em></strong></blockquote><p><strong>GO:</strong> The <a href="https://github.com/opendatadiscovery/opendatadiscovery-specification">ODD Specification</a> we designed treats various types of data equally. Tabular data, text, images, audio — for us, all of them are simply datasets in the system. They differ only in what metadata they contain. This allows our platform to build proper lineage no matter what type of dataset is being used.</p><p>Speaking of tabular data, the system should support column-based lineage in addition to object lineage. From a data compliance regulations standpoint, it’s important to understand where and how the data is used.</p><p>Also, it’s important to collect all the data profiling to eliminate black-box issues with anomaly detection. You need to know that your ML pipelines are working correctly. This is where the real end-to-end lineage helps — without it, black-box issues, including anomalies, are impossible to detect.</p><blockquote><strong><em>The MLOps technology space is becoming increasingly fragmented, particularly in areas like data observability. Do you believe data observability will remain as a standalone category, or will it become a feature of larger ML platforms?</em></strong></blockquote><p><strong>GO:</strong> The MLOps space is very fragmented because its formation is still in the early stages. Its development is very similar to the data world, although there are some differences between them, and ML operates different sets of entities. But the operations are very similar. My view is that it will eventually become unified and commodified in the same way it happened in the data world. All ML platforms will include end-to-end lineage for both data and ML, in the same way data platforms include monitoring. It will become a must-have built-in default feature, helping us stay on top of the black-box issues and ensuring that everything works correctly.</p><p>The current process of gathering metadata is mostly through proprietary products, but imposing vendor locks will shift to open standards. The protocol for gathering metadata should be open to eliminating processes that make both low-level data operations and data discovery very inefficient.</p><blockquote><strong><em>In recent years, we have seen some interesting research on applications of ML methods for streamlining different MLOps areas. Could that be the case for data discovery and observability? Which modern ML methods can be used to improve data discovery and observability in ML models in the near future?</em></strong></blockquote><p><strong>GO:</strong> Any data observability relies heavily on anomaly detection. In the world of ML, there are numerous methods for detecting anomalies. So there is a lot of space for ML to develop in the direction of data observability, to predict when there is something wrong with your data.</p><p>For efficient data discovery, it’s important to conduct a quick and personalized search. There can be thousands of similar datasets with minor differences, making them hard to choose from. We all have context surrounding us at work, and personalizing it will help boost productivity. We’re already seeing these types of ML models being used for recommendations in online shops, streaming services, and online search engines.</p><h3>Miscellaneous — a set of rapid-fire questions</h3><p>When talking about groups of data, it’s worth mentioning the Friendship paradox. Most people have fewer friends on average than all their friends have. This paradox is applicable to any graph data structure.</p><blockquote><strong><em>What book would you recommend to an aspiring ML engineer?</em></strong></blockquote><p>“ <a href="https://academic.oup.com/mind/article/LIX/236/433/986238">Computing Machinery and Intelligence</a> “ by Alan Turing. This paper laid the entire foundation for all of data science.</p><blockquote><strong><em>Is the Turing Test still relevant? Any clever alternatives?</em></strong></blockquote><p>I admire Turing too much to consider alternatives.</p><p>If there was a simple answer to this question, there would not be a Nobel Prize for it. When we can answer this question, the secret to math will be completely unveiled.</p><p>*This post is supported by Provectus. We thank the Provectus team for their ongoing support of TheSequence.</p><p><em>Originally published at </em><a href="https://thesequence.substack.com/p/osin"><em>https://thesequence.substack.com</em></a><em> on September 15, 2021.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8c3d9b2a28c3" width="1" height="1" alt=""><hr><p><a href="https://medium.com/opendatadiscovery/german-osin-provectus-about-data-discovery-and-observability-in-ml-solutions-8c3d9b2a28c3">🎙 German Osin/Provectus About Data Discovery and Observability in ML Solutions</a> was originally published in <a href="https://medium.com/opendatadiscovery">Open Data Discovery</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Overview of UI Tools for Monitoring and Management of Apache Kafka Clusters]]></title>
            <link>https://medium.com/data-science/overview-of-ui-tools-for-monitoring-and-management-of-apache-kafka-clusters-8c383f897e80?source=rss-6ed1e3b217d8------2</link>
            <guid isPermaLink="false">https://medium.com/p/8c383f897e80</guid>
            <category><![CDATA[ui]]></category>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[monitoring]]></category>
            <category><![CDATA[comparison]]></category>
            <category><![CDATA[management]]></category>
            <dc:creator><![CDATA[German Osin]]></dc:creator>
            <pubDate>Thu, 02 Sep 2021 18:21:56 GMT</pubDate>
            <atom:updated>2024-03-22T13:20:25.522Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nInAuRSd_5bjjyiTtNLvzg.png" /><figcaption>Image by <a href="https://ru.freepik.com/kanawatth">kanawatTH</a> from <a href="https://ru.freepik.com/premium-photo/digital-cyberspace-and-data-network-connections_9285251.htm">freepic.com</a></figcaption></figure><p>What are the best tools engineers can use to observe data flows, track key metrics, and troubleshoot issues in Apache Kafka?</p><p>Apache Kafka is an open-source distributed event streaming platform that enables organizations to implement and handle high-performance data pipelines, streaming analytics, data integration, and mission-critical applications. It is used by thousands of companies, 80% of which are Fortune 100 companies.</p><p>Though Apache Kafka is the go-to service for scenarios requiring real-time data processing and application activity tracking, the monitoring and management of clusters in Apache Kafka often pose a challenge. To make these tasks more efficient and transparent, you may need third-party, open-source, or commercial graphical tools that offer additional administrative and monitoring features.</p><p>This article provides an overview of such UI tools, including:</p><ol><li>AKHQ</li><li>Kowl</li><li>Kafdrop</li><li>UI for Apache Kafka</li><li>Lenses</li><li>CMAK</li><li>Confluent CC</li><li>Conduktor</li></ol><p>But first, let’s delve a bit into the problems of observability and monitoring in Apache Kafka.</p><h3>Observability and Monitoring in Apache Kafka</h3><p>Apache Kafka is a critical service for any data-driven organization. But handling vanilla Apache Kafka clusters can be quite painful. Kafka clusters are difficult to set up, tricky to scale, and expensive to maintain. Most importantly, they are error-prone and complicated to observe, which may cause various business-critical issues in data streams and prevent engineers from tracking and solving them.</p><p>To reduce errors and avoid critical issues, it is important for organizations to foolproof their Apache Kafka clusters by adding a robust observability component. Better observability can help to:</p><ul><li>Troubleshoot data streams much faster</li><li>Improve collaboration among engineers by having a shared understanding of the data streams they manage through metadata.</li><li>Discover sensitive data within data streams more easily, to meet compliance requirements.</li><li>Clean data more quickly and efficiently. Fewer incorrect dashboards mean happier customers.</li></ul><p>The challenge with cluster observability is that Apache Kafka comes with CLI tools for all necessary administrative tasks. But because they are not integrated into a single service, you have to run different tools separately for different tasks, which causes inconvenience and time loss. The problem can escalate quickly as your clusters grow in size, or if you have multiple clusters, simply because you lack the ability to properly observe them.</p><p>This brings us back to UI monitoring tools that can help organizations simplify and accelerate development, minimize time to resolution, and speed up the reporting process, to improve operational efficiency within and between engineering teams.</p><h3>Top 8 UI Monitoring Tools for Apache Kafka Clusters</h3><p>To begin with, a quick comparison of the Apache Kafka cluster monitoring tools.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MZLPKPUqwyAGMc88k6IzSQ.png" /><figcaption>Image by Author</figcaption></figure><h4>AKHQ</h4><ul><li>GitHub: <a href="https://github.com/tchiotludo/akhq">https://github.com/tchiotludo/akhq</a></li><li>License: Apache 2</li><li>Availability: Free</li><li>Pros: A lot of useful features</li><li>Cons: Bad UI; Lacks KSQL integration; Partial support of Protobuf schema registry</li></ul><p><a href="https://akhq.io/">AKHQ</a> (previously known as KafkaHQ) is a Kafka GUI for Apache Kafka that gives engineering teams the ability to search and explore data in a unified console. With AKHQ, developers and DevOps can manage topics, topic data, consumer groups, schema registry, connectivity, and more.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mjTgqAy5yk-Kqn_lqtoAaQ.png" /><figcaption>Image by Author</figcaption></figure><p>AKHQ offers tons of useful features, including multi-cluster management, message browsing, live tailing, authentication, authorization, read-only mode, schema registry, and Kafka Connect management. It supports Avro and is compatible with LDAP and RBAC.</p><p>But AKHQ’s UI is not the most convenient. You will definitely need to allocate some time to scale the learning curve.</p><p>On top of that, AKHQ does not have KSQL integration and provides only partial support of the Protobuf schema registry. Nor does it work with Dynamic Topic Configuration, Partition Increase, Replica Change, Kafka Streams Topologies, or JMX metrics visualization and charts.</p><p>Those who are looking to use AKHQ as part of their data streaming solutions on AWS should know that it does not support <a href="https://docs.aws.amazon.com/msk/latest/developerguide/iam-access-control.html">AWS Identity and Access Management (IAM) Access Control for Amazon MSK</a>.</p><h4>Kowl</h4><ul><li>GitHub: <a href="https://github.com/cloudhut/kowl">https://github.com/cloudhut/kowl</a></li><li>License: Apache 2</li><li>Availability: Partially paid</li><li>Pros: Good UI</li><li>Cons: Lacks many features</li></ul><p><a href="https://cloudhut.dev/">Kowl</a> (previously known as Kafka Owl) is a web application designed to help developers explore messages in Apache Kafka clusters and gain better insights into what is actually happening in these clusters.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SFe4K7ao2pb26UKRL6bfTg.png" /><figcaption>Image by Author</figcaption></figure><p>The biggest advantage of Kowl is its fantastic UI. It is convenient, user-friendly, and quite straightforward to use. It does not come with a lot of features, though.</p><p>For instance, Kowl offers message browsing, live tailing, and support of Protobuf, Avro, and Amazon MSK IAM, but login system (Google, GitHub, Okta) and RBAC permissions with group syncing is available only with a paid Kowl Business plan.</p><p>Kowl is also missing such features as multi-cluster management, dynamic topic configuration, partition increase, replica change, Kafka Connect management, schema registry, KSQL integration, Kafka Streams topologies, read-only mode, and visualization and charts for JMX metrics. If these were included in the package, Kowl would be preferable to any other tool.</p><h4>Kafdrop</h4><ul><li>GitHub: <a href="https://github.com/obsidiandynamics/kafdrop">https://github.com/obsidiandynamics/kafdrop</a></li><li>License: Apache 2</li><li>Availability: Free</li><li>Pros: Nice community</li><li>Cons: Average UI; Lacks a lot of features</li></ul><p><a href="https://github.com/obsidiandynamics/kafdrop">Kafdrop</a> is a web UI for viewing Apache Kafka topics and browsing consumer groups. The tool enables developers to more easily display and handle cluster information such as brokers, topics, partitions, and consumers. It also allows for viewing messages.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kciaM0yMkkSTXVKLjxIsQg.png" /><figcaption>Image by Author</figcaption></figure><p>For the most part, Kafdrop is a pretty average tool. Its UI is not spectacular, and it lacks a lot of features. It does allow you to view Kafka brokers and consumer groups, create and view topics, browse messages, and monitor ACLs. It also provides support for Azure Event Hubs. But what about other useful features like live tailing, schema registry, or read-only mode?</p><p>The good news is that Kafdrop is highly rated on GitHub, and if you are looking for a helpful and immersive community, this may well be the tool for you.</p><h4>UI for Apache Kafka</h4><ul><li>GitHub: <a href="https://github.com/kafbat/kafka-ui">https://github.com/kafbat/kafka-ui</a></li><li>License: Apache 2</li><li>Availability: Free</li><li>Pros: Good UI; Flexible; Lots of features</li><li>Cons: Development in progress</li></ul><p><a href="https://github.com/kafbat/kafka-ui">UI for Apache Kafka</a> is a free open-source web service that offers developers a clear UI for handling Apache Kafka clusters. It enables developers to monitor data flows and find and troubleshoot issues in data while delivering optimal performance. The lightweight dashboard makes it easy to keep track of key metrics of Apache Kafka clusters, including Brokers, Topics, Partitions, Production, and Consumption.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1002/0*kVCWWgZrmoXuTPY2" /><figcaption>Image by Author</figcaption></figure><p>UI for Apache Kafka stands out from the crowd thanks to its convenient UI, free availability, and sheer number of features. It boasts such capabilities as:</p><ul><li><strong>Browse Messages</strong> — browse messages with Avro, Protobuf, JSON, and plain text encoding</li><li><strong>View Consumer Groups</strong> — view per-partition parked offsets, and combined and per-partition lag</li><li><strong>Configurable Authentification</strong> — secure your installation with optional Github/Gitlab/Google OAuth 2.0</li><li><strong>View Kafka Brokers</strong> — view topic and partition assignments, and controller status</li><li><strong>View Kafka Topics</strong> — view partition count, replication status, and custom configuration</li><li><strong>Multi-Cluster Management</strong> — monitor and manage all your clusters in one place</li><li><strong>Dynamic Topic Configuration</strong> — create and configure new topics with dynamic configuration</li></ul><p><a href="https://provectus.com/">Provectus</a>, the AI consultancy that is designing and building the tool, claims that it will add more features, including live tailing, KSQL integration, Kafka Streams topologies, and JMX metrics visualization and charts, as early as the end of August.</p><h4>Lenses</h4><ul><li>GitHub: <a href="https://github.com/lensesio">https://github.com/lensesio</a></li><li>License: BSL</li><li>Availability: Free</li><li>Pros: Awesome with fast-kafka-dev and for local development</li><li>Cons: Lacks many features</li></ul><p><a href="https://lenses.io/">Lenses</a> positions itself as a DataOps platform for real-time applications and data operations for Apache Kafka and Kubernetes. It can help engineers make data more usable and secure, and eliminate data silos. Lenses appear to be the highest-rated product for real-time stream analytics.</p><p>But Lenses is arguably a fairly average tool. It makes perfect sense to use Lenses with fast-kafka-dev. It is also good for local development. However, it lacks certain features; multi-cluster management, message browsing, and avro support are simply not enough to make it work for many tasks. Offering Kafka Connect management as a separate service does not help, either.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dqHFO7LyJWvdEsKXilfH5w.png" /><figcaption>Image by Author</figcaption></figure><p>But if you are content without a lot of features, Lenses’s UI will absolutely fit the bill. It is a truly stunning tool that is very sleek and intuitive.</p><h4>CMAK</h4><ul><li>GitHub: <a href="https://github.com/yahoo/CMAK">https://github.com/yahoo/CMAK</a></li><li>License: Apache 2</li><li>Availability: Free</li><li>Pros: Great for partition reassignment; Ops tool</li><li>Cons: Limited to Ops tasks</li></ul><p><a href="https://github.com/yahoo/CMAK">CMAK</a> (previously known as Kafka Manager) is a comprehensive tool that enables engineers to manage Apache Kafka clusters for various Ops tasks.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oWCZHMD5rmOjYQDwrruolA.png" /><figcaption>Image by Author</figcaption></figure><p>CMAK boasts a good and fairly straightforward UI. Though it does not provide a lot of features, multi-cluster management, dynamic topic configuration, partition creation, and replica change will cover most of your tasks.</p><p>For the most part, CMAK is primarily an Ops tool. It is also really good at partition reassignment.</p><h4>Confluent CC</h4><ul><li>GitHub: <a href="https://github.com/confluentinc">https://github.com/confluentinc</a></li><li>License: Paid</li><li>Availability: Paid</li><li>Pros: Part of Confluent Enterprise</li><li>Cons: Part of Confluent Enterprise</li></ul><p><a href="https://www.confluent.io/product/confluent-platform/gui-driven-management-and-monitoring/">Confluent Control Center</a> is a web-based user interface that allows developers and operators to manage and monitor Apache Kafka clusters, including checking cluster health, observing and controlling messages, topics, and Schema Registry. It can also be used to develop and run ksqlDB queries.</p><p>The important thing about Confluent CC is that it is offered as part of Confluent Enterprise, meaning it is a paid service only. It comes with lots of features and a really good user interface. If you don’t mind being locked into the Confluent ecosystem, this UI tool will more than meet your needs.</p><p>Overall, Confluent CCS is more than just an ordinary topic checking tool. Its features are superb, and all of them work perfectly, without any glitches.</p><h4>Conduktor</h4><ul><li>GitHub: <a href="https://github.com/conduktor">https://github.com/conduktor</a></li><li>License: Paid</li><li>Availability: Paid</li><li>Pros: Tons of features</li><li>Cons: Desktop tool</li></ul><p><a href="https://www.conduktor.io/">Conduktor</a> is a desktop client for Apache Kafka that provides engineers with a user-friendly interface to work with the Kafka ecosystem. The client is native to Windows, Linux, and Mac. It can handle any type of Apache Kafka cluster and boasts every feature you could ask for.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*d7DYuDe7hzUe24ce3kju5w.png" /><figcaption>Image by Author</figcaption></figure><p>But Conduktor may not be as convenient as other UI tools in this list, because it is a desktop application. If you are OK with that, Conduktor can be a viable alternative to Confluent CC.</p><h3>Conclusion</h3><p>Having the right UI tools for monitoring and management of Apache Kafka clusters is key to cluster health. With a simple and convenient user interface, you can observe data flows, track metrics, and troubleshoot issues more efficiently, without having to use dozens of additional CLI tools. This results in fewer bottlenecks, faster reporting, and more cost-efficient development.</p><p>This article offers my perspective on major UI tools for monitoring and management of clusters in Apache Kafka. I did my best to be objective, but the community is sure to have something to add.</p><p>Feel free to share your feedback and opinions about these UI tools for Apache Kafka, and my overview of them, in the comment section.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8c383f897e80" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/overview-of-ui-tools-for-monitoring-and-management-of-apache-kafka-clusters-8c383f897e80">Overview of UI Tools for Monitoring and Management of Apache Kafka Clusters</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Feature Store as a Foundation for Machine Learning]]></title>
            <link>https://medium.com/data-science/feature-store-as-a-foundation-for-machine-learning-d010fc6eb2f3?source=rss-6ed1e3b217d8------2</link>
            <guid isPermaLink="false">https://medium.com/p/d010fc6eb2f3</guid>
            <category><![CDATA[editors-pick]]></category>
            <category><![CDATA[mlops]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[making-sense-of-big-data]]></category>
            <dc:creator><![CDATA[German Osin]]></dc:creator>
            <pubDate>Thu, 10 Dec 2020 13:17:18 GMT</pubDate>
            <atom:updated>2020-12-15T22:21:28.112Z</atom:updated>
            <content:encoded><![CDATA[<h4><a href="https://towardsdatascience.com/tagged/making-sense-of-big-data">Making Sense of Big Data</a></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lz7Kj4lz5AM2uHvTAJ5KHQ.jpeg" /><figcaption>Image by <a href="https://www.shutterstock.com/ru/g/Yurchanka+Siarhey">Yurchanka Siarhei </a>from <a href="https://www.shutterstock.com/ru/image-illustration/artificial-digital-neural-network-blockchain-technology-1046678647">shutterstock</a></figcaption></figure><p>Artificial intelligence and machine learning have reached an inflection point. In 2020, organizations in diverse industries of various sizes began evolving their ML projects from experimentation to production on an industrial scale. While doing so, they realized they were wasting a lot of time and effort on feature definition and extraction.</p><p>Feature store is a fundamental component of the ML stack, and of any robust data infrastructure, because it enables efficient feature engineering and management. It also allows for simple reuse of features, feature standardization across the company, and feature consistency between offline and online models. A centralized, scalable feature store allows organizations to innovate faster and drive ML processes at scale.</p><p>The team at <a href="https://provectus.com/">Provectus</a> has built several feature stores, and we would like to share our experience and lessons learned. In this article, we will explore how feature stores can help eliminate rework and enforce data traceability from source to model across teams. We will look into the specifics of building a scalable feature store, and discuss how to achieve consistency between real-time and training features, to improve reproducibility with time traveling for data.</p><h3>What Is a Feature Store?</h3><p>A Feature Store is a data management layer for machine learning features. ML features are measurable properties of phenomena under observation, like raw words, pixels, sensor values, rows of data in a data store, fields in a CSV file, aggregates (min, max, sum, mean), or derived representations (embedding or cluster).</p><p>From a business perspective, feature stores offer two major benefits:</p><ol><li><strong>Better ROI from feature engineering through reduction of cost per model,</strong> which facilitates collaboration, sharing, and reusing of features</li><li><strong>Faster time to market for new models, driven by increased productivity of ML Engineers</strong>. This enables organizations to decouple storage implementation and features serving API from ML engineers, freeing them to work on models, not on latency problems, for more efficient online serving.</li></ol><p>Some use cases do not require a centralized, scalable feature store. Feature stores work for projects where models need a stateful, ever-changing representation of the system. Examples of such use cases include demand forecasting, personalization and recommendation engines, dynamic pricing optimization, supply chain optimization, logistics and transportation optimization, fraud detection, and predictive maintenance.</p><h3>Concepts of a Feature Store</h3><p>A standardized feature store has certain key concepts that revolve around it. Those concepts are:</p><ol><li><strong>Online Feature Store</strong>. Online applications look for a feature vector that is sent to an ML model for predictions</li></ol><p>2. <strong>ML Specific Metadata</strong>. Enables the discovery and reuse of features</p><p>3. <strong>ML Specific API and SDK</strong>. High-level operations for fetching training feature sets and online access</p><p>4. <strong>Materialized Versioned Datasets</strong>. Maintains versions of featuresets used to train ML models</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Qgx3EW5E_7VeaioH" /><figcaption>Image by Author</figcaption></figure><p>All of the concepts are represented in the above image. Analytical data is taken from a data lake and pushed through the feature engineering pipeline, where output is stored in the feature store. From there, ML engineers can discover the features, use them to train new models, and then re-use the features for serving.</p><p>These four concepts are supported by multiple products. The leaders of market are Feast, Tecton, Hopsworks and <a href="https://aws.amazon.com/ru/sagemaker/feature-store/">AWS SageMaker Feature Store</a>. We will focus on open source products: Feast and Hopsworks.</p><h4>#1 Feast</h4><p><a href="https://feast.dev/">Feast</a> is an ML service that helps teams bridge the gap between data and machine learning models. It allows teams to register, ingest, serve, and monitor features in production.</p><p>This service is in the active development stage, but it has already been battle-tested with GoJek, Farfetch, Postmates, and Zulily. It can be integrated with Kubeflow and is backed by Kubeflow’s strong community.</p><p>As of 2020, Feast is a GCP-only service. It is a bit infrastructure-heavy, lacks composability, and does not offer data versioning. Note that the company plans to address these challenges in its roadmap for 2021. And starting November 2020, Feast is available as an <a href="https://www.tecton.ai/blog/feast-announcement/">open source version of Tecton</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/811/0*6oYo0-1x8eKLO_w2" /><figcaption>Image by Author</figcaption></figure><h4>#2 Hopsworks</h4><p><a href="https://www.logicalclocks.com/">Hopsworks</a> is an Enterprise Platform for developing and operating AI applications. It allows teams to manage ML features quickly and efficiently. The team behind Hopsworks are feature store evangelists, and they offer a lot of great educational content.</p><p>The Hopsworks feature store can be integrated with most Python libraries for ingestion and training. It also supports offline stores with time travel. Most importantly, it is AWS-, GCP-, Azure-, and on-premise ready.</p><p>What makes Hopsworks a challenge to use is its heavy reliance on the HopsML infrastructure. Also, the Hopsworks online store may not satisfy all latency requirements.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/810/0*j_ny0fq_QhlowMz1" /><figcaption>Image by Author</figcaption></figure><h3>Challenges of Modern Data Platforms</h3><p>Before looking into the specifics of building feature stores, the challenges of modern data platforms have to be considered. Feature stores cannot be examined in isolation from the rest of the data and the <a href="https://provectus.com/machine-learning-infrastructure/">ML infrastructure</a>.</p><p>Traditionally, a canonical data lake architecture looks like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*w1l6Y7src7ufa7F1" /><figcaption>Image by Author</figcaption></figure><p>This high-level architecture has components for sourcing, ingestion, processing, persisting, querying, and a consumer component. While it works fine for most tasks, it is not ideal.</p><p><strong>Data access and discoverability</strong> can be a problem. Data is scattered across multiple data sources and services. This once helped protect data, but now it only adds a new layer of complexity and creates data silos. Such architecture entails the tedious process of managing AWS IAM roles, Amazon S3 policies, API Gateways, and database permissions. This becomes even more complicated in a multi-account setup. As a result, engineers are confused about what data exists, and which is actually accessible to them, since metadata is not discoverable by default. This creates an environment where investments in data and machine learning are curtailed due to data access issues.</p><p><strong>Monolithic data teams</strong> are another issue to consider. Since data and machine learning teams are extremely specialized, they often have to operate out of context. Lack of ownership and domain context creates silos between data producers and data consumers, making it considerably more difficult for a backlogged data team to keep up with business demands. Data and ML Engineering fall victim to complex dependencies, failing to sync their operations. Any fast, end-to-end experimentation is not possible in such circumstances.</p><p><strong>Machine learning experimentation infrastructure</strong> is an uncharted territory. Traditional architecture lacks an experimentation component, which not only leads to data discovery and data access issues, but also makes it more complex to maintain the reproducibility of datasets, ML pipelines, ML Environments, and offline experiments. Although Amazon SageMaker and Kubeflow have made some progress here, reproducibility is problematic. Production experimentation frameworks are immature and cannot be entirely relied upon. As a result, it can take from three to six months to push one end-to-end experiment from data to production ML.</p><p><strong>Scaling ML in production</strong> is complex and requires a lot of time and effort. While machine learning is mostly discussed as an offline activity (e.g. data collection, processing, model training, evaluating results, etc.), the ways models are used and served online in the real world are what actually matter. With traditional architecture, you cannot access features during model serving in a unified and consistent way. You also cannot reuse features between multiple training pipelines and ML applications. The monitoring and maintenance of ML applications is also more complicated. As a result, the time and cost required to scale from 1 to 100 models in production grows exponentially, which limits the ability of organizations to innovate at the desired pace.</p><h3>Emerging Architectural Shifts</h3><p>To address these challenges, several architectural shifts have emerged:</p><ol><li><strong>From Data Lakes to Hudi/Delta lakes</strong>. A data lake is not just a folder in Amazon S3. It is a feature-rich, fully managed layer for Data Ingestion, Incremental Processing, and Serving with ACID transactions and point-in-time queries.</li><li><strong>From Data Lakes to Data Mesh</strong>. Ownership of data domains, data pipelines, metadata, and API is moving away from centralized teams to product teams. Another impactful benefit is the treating and owning data as a complete product, rather than a side effect that nobody cares about.</li><li><strong>From Data Lakes to Data Infrastructure as a Platform</strong>. If ownership of data is decentralized, platform components have to be unified and packaged into a reusable data platform.</li><li><strong>From Endpoint Protection to Global Data Governance</strong>. As part of the shift towards centralized data platforms, organizations are moving away from Endpoint Protection to GLobal Data Governance, which is a higher level control plan for managing security and data access policies across available data sources.</li><li><strong>From Metadata Store to Global Data Catalog</strong>. Metadata stores like Hive meta store cannot aggregate many data sources. The industry needs a Global Data Catalog to support User Experience around data discovery, lineage, and versioning.</li><li><strong>Feature Store</strong>. Feature store is a new emerging component of the ML stack that enables scaling of ML Experimentation and Operations by adding a separate data management layer for ML Features.</li></ol><p>All of these transformations are happening in parallel and should be thought of holistically. You cannot start designing a feature store and end up having separate data catalogs for features and for other data applications. While building a feature store, you have to rely on data versioning features, which can easily be part of other parallel projects.</p><p>Before moving forward, just a few words about four major components that drive the aforementioned transformations from disjointed ML jobs to MLOps, to provide a wider context for the importance of feature stores.</p><h4>#1 Delta/Hudi Lakes</h4><p>ACID data lakes enable managed ingestion, efficient dataset versioning for ML training, cheap “deletes” to make it GDPR / CCPA compliant, and “upserts” for data ingestions. They also offer an audit log to keep track of dataset changes and ACID transactions while enforcing data quality through schemas. Delta and Hudi lakes bring stream processing to Big Data, providing fresh data much more efficiently than traditional batch processing.</p><h4>#2 Global Data Governance</h4><p>Because it is no longer a standard to manage AWS IAM roles, Amazon S3 policies, API Gateways, and database permissions at the user level, a company-wide data governance structure should be used to:</p><ol><li><strong>Accelerate privacy operations with data you already have.</strong> Automate business processes, data mapping, and PI discovery and classification for privacy workflows.</li><li><strong>Operationalize policies in a central location.</strong> Govern privacy policies to ensure policies are effectively managed across the enterprise. Define and document workflows, traceability views, and business process registers.</li><li><strong>Scale compliance across multiple regulations.</strong> Use a platform designed and built with privacy in mind that can be easily extended to support new regulations.</li></ol><h4>#3 Global Data Catalog</h4><p>Although there is no single market leader here, Marquez, Amundsen, Apache Atlas, Collibra, Alation and Data Hub are worth mentioning.</p><p>A global data catalog is extremely useful for answering questions such as, Does this data exist? Where is it? What is the source of truth of this data? Do I have access? Who is the owner? Who are the users of this data? Are there existing assets I can reuse? Can I trust this data? Basically, it acts as a meta metadata store of sorts.</p><h4>#4 Reproducible ML Pipelines</h4><p>The final component is reproducible ML pipelines for experimentation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*JISC-wWJm5IReqYn" /><figcaption>Image by Author</figcaption></figure><p>The above chart represents the architecture for MLOps and reproducible experimentation pipelines. It starts with four inputs: ML model code, ML pipeline code, Infrastructure as a code, and Versioned dataset. The versioned dataset — an input for your machine learning pipeline — should be sourced from the feature store.</p><h3>Modern Data Infrastructure</h3><p>Now let’s look at modern data infrastructure.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*qH85zfTekxVOmej6" /><figcaption>Image by Author</figcaption></figure><p>We have batch processing for raw data and stream processing for event data. We store processed artifacts in cold storage for business reports and in a near-real-time, incrementally updated hot index for our API. The same data can be used in both scenarios and, to make it consistent, we use different pub/sub systems.</p><p>This is the traditional architecture for data platforms. Its goal is to provide consistency between cold and hot storage, and to allow for discoverability from the data catalog, data quality, and global security with a fine-grained control on top of it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/786/0*30CNRALU9sIa1Lf3" /><figcaption>Image by Author</figcaption></figure><p>If we look at a feature store design, we will see features and infrastructure components almost identical to what we have in the data platform. In this case, the feature store is not a separate silo that brings yet another ingestion system, storage, catalog and quality gates. It serves as a lightweight API between our data platform and ML tools. It can be nicely integrated with everything that has already been done in your data infrastructure. It should be composable and lightweight, and without an opinionated design.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/765/0*UZm5uzXd6PvrUJe-" /><figcaption>Image by Author</figcaption></figure><p>When you begin designing and building your data infrastructure, consider the following “lessons learned” so far:</p><ol><li>Start by designing a consistent ACID Data Lake before investing in a Feature Store</li><li>Value from existing open source products does not justify investments in integration and the dependencies they bring</li><li>A feature store is not a new infrastructure and data storage solution, but a lightweight API and SDK integrated into your existing data infrastructure.</li><li>Data Catalog, Data Governance, and Data Quality components are horizontal for the entire Data Infrastructure, including the feature store</li><li>There are no mature open source or cloud solutions for Global Data Catalog and Data Quality monitoring.</li></ol><h4>Reference Architecture</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/744/0*8Iw-NoHEr6Frqz2d" /><figcaption>Image by Author</figcaption></figure><p>This chart depicts the reference architecture we have been using for our customers. It features the services we have opted to use, but you should not be limited by our choices. The idea here is that you have to choose cold <em>and</em> hot storage based on your data workloads, and on your business needs.</p><p>For hot storage, you may choose from DynamoDB, Cassandra, Hbase, or traditional RDBMS like Mysql, PostgreSQL or even Redis. It is important that your hot storage be composable, pluggable, and in alignment with your data infrastructure strategy</p><p>For cold storage, Apache Hudi and Delta lake are our favorites. They offer such features as Time Travel, Incremental ingestion, and Materialized views.</p><p>There are some blank spaces on the diagram which we hope to fill soon. For example, so far there is no overshelf leader for the data catalog. Data quality tools are also in the early stages. For now, you can choose from Great Expectations or Apache Deequ, which are great tools, but they do not provide a complete solution.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/888/0*AtFYXqz5ZWGT04AI" /><figcaption>Image by Author</figcaption></figure><p>In the image above, the question marks occupy spaces where you can choose from solutions built by open source communities, build your own solution in-house, or collaborate with cloud providers (e.g. AWS’ latest addition — <a href="https://aws.amazon.com/ru/sagemaker/feature-store/">Amazon SageMaker Feature Store for Machine Learning</a>).</p><h3>Moving Forward with Feature Store</h3><p>Although it is still early in the game for feature stores, organizations that are not just experimenting but actively moving machine learning projects to production have already realized the need to have a centralized repository to store, update, retrieve, and share features.</p><p>In this article, we have shown how to design and build such a repository. While some of the points featured here are debatable and open for feedback from the community, it is clear that:</p><ol><li>Your existing data infrastructure should cover at least 90% of feature store requirements, including streaming ingestion, consistency, data catalog, and versioning, to achieve the desired outcome.</li><li>It makes sense to build a lightweight Feature Store API to integrate with your existing storage solutions in-house.</li><li>You should collaborate with community and cloud vendors to maintain compatibility with standards and state-of-the-art ecosystems.</li><li>You should be ready to migrate to a managed service, or to an open source alternative as the market matures.</li></ol><p>Feel free to share your vision of feature stores. We are open to discussion.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d010fc6eb2f3" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/feature-store-as-a-foundation-for-machine-learning-d010fc6eb2f3">Feature Store as a Foundation for Machine Learning</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>