<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Provectus - Medium]]></title>
        <description><![CDATA[Provectus is an Artificial Intelligence consultancy and solutions provider, helping businesses achieve their objectives through AI. - Medium]]></description>
        <link>https://medium.com/provectus?source=rss----b886f77f9ec8---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>Provectus - Medium</title>
            <link>https://medium.com/provectus?source=rss----b886f77f9ec8---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 14:01:33 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/provectus" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Finding the Right Data Catalog Solution]]></title>
            <link>https://medium.com/provectus/finding-the-right-data-catalog-solution-a265a4b3c0c3?source=rss----b886f77f9ec8---4</link>
            <guid isPermaLink="false">https://medium.com/p/a265a4b3c0c3</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[comparison]]></category>
            <category><![CDATA[data-catalog]]></category>
            <category><![CDATA[data-observability]]></category>
            <dc:creator><![CDATA[German Osin]]></dc:creator>
            <pubDate>Mon, 12 Sep 2022 14:15:53 GMT</pubDate>
            <atom:updated>2022-04-06T15:16:27.171Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*P0VZR1Cd7J6Rgkxy.png" /><figcaption>Image by <a href="https://ru.freepik.com/alexeybrin">alexey-brin</a> from <a href="https://ru.freepik.com/premium-photo/3d-render-of-abstract-shape-that-chaotically-deformed-data-centralisation-concept_17405594.htm">freepic.com</a></figcaption></figure><p>When organizations begin to adopt artificial intelligence, machine learning, and big data analytics, they realize that none of these technologies offers a magical fix that can change everything about their business overnight. In fact, to start driving actual value, they require fundamental, long-term changes in the organization’s culture, technology setting, and operating model. One key component that can enable a successful transformation is a data catalog, a management service that organizes your data in one place and lets you tag it with metadata, so you can discover and govern it more efficiently.</p><p>As someone who has been helping businesses to design and build robust solutions for handling data, I am going to share my experience about some of the most common data catalogs, to help you choose the right data catalog solution for your organization.</p><p>To begin, let’s take a closer look at data catalogs and the challenges they help resolve.</p><h3>Data Catalogs and the Challenge of Data Discovery and Observability</h3><p>Today, every business owner understands the importance of data, and the valuable insights it provides about your company and your customers. However, with an explosion in the volume of data, finding the right data has become harder than ever before. You can easily collect data, but what about further steps like data processing, data analysis, and BI? And what about data access, data discovery, data observability, and data governance?</p><p>In truth, a significant number of tech-savvy organizations, not to mention early adopters of any data- and intelligence-driven initiatives, find it challenging to understand the data they collect, account for how the data is moved around, and explain how and for what purpose data is used by different business divisions.</p><p>Unfortunately, the correct handling of data poses many challenges. These include:</p><ul><li>Difficulty finding and accessing the right data</li><li>Ineffective data collection: data is dumped vs being organized</li><li>Lack of understanding of the importance of data across the organization</li><li>Inefficient data processing: data lacks structure and is hard to understand</li><li>Inefficiencies in assessing the quality and trustworthiness of data in data swamps</li><li>Lack of infrastructure to capture, enhance, and reuse data assets in the organization</li><li>Reliance on manual and ad-hoc data preparation efforts</li></ul><p><strong>In short:</strong> Data has become too complicated. It takes increasingly more time for professionals to collect, process, analyze, and manage data efficiently and at scale.</p><p>To make matters worse, existing data catalog solutions do not entirely cover the needs of IT organizations and businesses. They lack standardized data collection methods, are incompatible with different catalogs, provide limited data lineage, and rely on inefficient data quality and observability practices.</p><p>As a result, data scientists spend lots of time discovering available datasets and making sure they can be trusted. Data science and ML engineering teams allocate about 90% of their time to ensuring that data is clean and reliable, and to tuning, debugging, and maintaining data pipelines that power mission-critical analytical dashboards and ML models. Enterprises are forced to hire armies of data stuarts, just to catalogize and curate data sets.</p><p>The challenge of data discovery, reliability, observability, and governance is real. The leading data catalogs are doing their best to make a difference in this arena. Below are some data catalogs that I can recommend for use in data- and intelligence-driven organizations.</p><h3>Top Data Catalogs for Enterprises</h3><p>Let’s kick off with a comparison table of open-source data catalog solutions, and continue by looking into each of them in more detail. Bear in mind that I have explored almost 30 different data catalog vendors, and narrowed it down to those featured below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*LjNDBWbMN3kBgQFe.png" /><figcaption>Image by author</figcaption></figure><h4>Amundsen</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*IcnjeolDipDvmUnR" /><figcaption>Image by author</figcaption></figure><ul><li>Pros: open-source, good community, extensive documentation</li><li>Cons: moderate UI, lack of ML entities</li></ul><p><a href="https://www.amundsen.io/">Amundsen</a> is an open-source data discovery and metadata engine that is designed and built to meet various needs of analysts and data scientists on one hand, and data engineers and software engineers on the other. The team claims that their solution can make data engineers, data analysts, and data scientists 20% more productive.</p><p>Amundsen has some of the best documentation, and developers call it the “Google of data cataloging.” It ranks data based on usage patterns. The UI itself can be characterized as moderate, since it does not provide space and cost metrics. It does, however, provide column statistics and a data preview. It supports Apache Hive, Amazon Redshift, Apache Druid, RDBMS, Presto, Snowflake, and several others listed on their GitHub page.</p><p><strong>Note: </strong>Amundsen has an SaaS version called <a href="https://www.stemma.ai/">Stemma</a>. It is a fully managed data catalog on its own, and features lots of useful integrations for data warehouses, orchestration and transformation, business intelligence, collaboration, security, and HR systems.</p><h4>Datahub</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*nkPX2uE0SdWeaooW" /><figcaption>Image by author</figcaption></figure><ul><li>Pros: open-source, good UI, lots of useful features (e.g. glossary and domains)</li><li>Cons: hard to install, custom ingestion protocol</li></ul><p><a href="https://linkedin.github.io/datahub/">Datahub</a> is an open-source project built by the people at LinkedIn. It has push and pull features, and its UI is modern and easy to use. The main issue with this data catalog is that it requires a lot of infrastructure effort, making it difficult to implement for smaller teams. It does not provide column statistics or metrics, and one minor flaw is that it supports only limited sources including Apache Hive, Apache Kafka, and RDBMS.</p><p>Datahub has an excellent <a href="https://demo.datahubproject.io/">demo</a> that lets you try out the product at a smaller scale. It also has an SaaS version called <a href="https://metaphor.io/">Metaphor</a> — a modern, scalable, and cloud-first data catalog and discovery service.</p><h4>Open Data Discovery</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*fIro2HwkINvea6KA" /><figcaption>Image by author</figcaption></figure><ul><li>Pros: good UI, ingestion based on specification</li><li>Cons: some features still in development</li></ul><p><a href="https://github.com/opendatadiscovery/odd-platform">Open Data Discovery (ODD)</a> is an open-source data discovery and observability platform. It is designed to help data-driven enterprises democratize data by making it more discoverable, manageable, observable, reliable, and secure. Because it supports <a href="https://github.com/opendatadiscovery/opendatadiscovery-specification">open data standards</a>, ODD enables data teams to make more efficient data exchanges between various data tools.</p><p>The UI of the platform is pretty good, and its ingestion is based on specification. Bear in mind that the platform is a work in progress, so some of the features are still in development.</p><h4>Apache Atlas</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*-725jvMMQXIJpzTC" /><figcaption>Image by author</figcaption></figure><ul><li>Pros: open-source, great metadata model</li><li>Cons: complex UI, a bit outdated</li></ul><p><a href="https://atlas.apache.org/#/">Apache Atlas</a> is an open metadata management and governance ecosystem that enables enterprises to meet their compliance requirements within Apache Hadoop. By supporting many integrations for classifying and governing data, it allows organizations to build a scalable catalog featuring various data assets, and to provide collaboration capabilities around those data assets for data scientists, analysts, and engineers.</p><p>All of this makes Apache Atlas a convenient tool for handling metadata. The challenge is that its ecosystem is a bit outdated (e.g. historically, a legacy product) and comes with a complex UI for advanced users.</p><h4>Marquez</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*5W8ACkM54gwOlc6y" /><figcaption>Image by author</figcaption></figure><ul><li>Pros: open-source, specification-based ingestion protocol, minimal infrastructure requirements to run</li><li>Cons: moderate UI, only a couple of adapters</li></ul><p>Originally created by WeWork, <a href="https://marquezproject.github.io/marquez/">Marquez</a> is an open-source metadata service that enables organizations to collect, aggregate, and visualize a data ecosystem’s metadata. It allows for the simplification of flexible data lineage queries across all datasets. It supports push functionality and works with Amazon S3 applications.</p><p>Marquez comes with a basic UI and only a couple of adapters. But these disadvantages are compensated for by a convenient specification-based ingestion protocol. The service does not require advanced infrastructure to run.</p><p><strong>Note:</strong> <a href="https://datakin.com/">Datakin</a> is an SaaS version of Marquez. It supports Apache Hive, Amazon RDS, Teradata, Amazon Redshift, Amazon S3, and Cassandra.</p><h4>Magda</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*X5LhWgawDkeYNbj_" /><figcaption>Image by author</figcaption></figure><ul><li>Pros: open-source, good for geodata, in-depth documentation</li><li>Cons: lack of adapters for Big Data solutions</li></ul><p><a href="https://magda.io/">Magda</a> positions itself as a federated open-source data catalog for all your big data and small data (e.g. Excel and CSV files). One area where Magda excels is geospatial and location data. If your organization works with geolocations, Magda might be the best option for you. It supports push features using its UI, and aside from Amundsen, is one of the few that support a data preview. But bear in mind that lack of adapters for big data can be a limiting factor.</p><h4>CKAN</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*CNP7iH7z91S90UOt" /><figcaption>Image by author</figcaption></figure><ul><li>Pros: open-source, thorough documentation</li><li>Cons: lack of adapters for Big Data solutions</li></ul><p><a href="https://ckan.org/">CKAN</a> is an open-source data management system for data hubs and portals. It provides capabilities for cataloging, storing, and accessing datasets with an API, featuring tools for handling data, data catalog, visualization tools, etc. CKAN is known for its rich documentation and support. However, the tool lacks some adapters for big data.</p><h4>OpenMetadata</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*KdjjLAdtZCWzgWys" /><figcaption>Image by author</figcaption></figure><ul><li>Pros: open-source, good UI, fast release cycles</li><li>Cons: complex installation, custom ingestion protocol, lack of ML entities</li></ul><p><a href="https://open-metadata.org/">OpenMetadata</a> is an open standard for metadata that provides a foundation for an end-to-end metadata management solution. It includes all necessary components for data discovery and governance, data quality, and observability. In combination with its quality UI, these features ensure ease of use and fast release cycles.</p><p>There are some key challenges with OpenMetadata, however. For instance, it is quite hard to install. It supports a custom ingestion protocol. And most importantly, it does not include entities for machine learning, which can be a decisive factor for data- and AI-driven enterprises.</p><h4>Atlan</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*pCbgvDqsjR-E-SJ4" /><figcaption>Image by author</figcaption></figure><ul><li>Pros: decent UI, well-funded startup, integrated data quality</li><li>Cons: custom ingestion protocol (requires access to your data from Atlan cloud), lack of ML entities</li></ul><p><a href="https://atlan.com/">Atlan</a> describes itself as a modern data workspace that helps data teams collaborate and democratize their enterprise data, while automating repetitive tasks. Its platform consists of three major components: data catalog and discovery, data lineage and governance, and data exploration and integrations. One of the perks of Atlan is that it features an integrated component for data quality assurance. Clean and unbiased data is key to advanced analytics and AI/ML projects.</p><p>One aspect where Atlan lags behind is its custom ingestion protocol. You will need to access your data from Atlan cloud to start working with it. It also lacks some ML entities.</p><p>Atlan supports Presto, Deequ, Atlas, Airflow, and Hudi.</p><h4>Alation</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*kktVRgiNI6dXi3mr" /><figcaption>Image by author</figcaption></figure><ul><li>Pros: proper UI, lots of great features like glossary, policies, etc.</li><li>Cons: custom ingestion protocol, lack of ML entities</li></ul><p><a href="https://www.alation.com/">Alation</a> is a data catalog that prioritizes the human aspect of handling data. It supports many useful features for faster onboarding, query re-use, and automated curation. On top of that, its UI is simple, consistent, and intuitive. If you are going to use Alation, however, bear in mind that it has a custom ingestion protocol and lacks ML entities. This is critical for businesses that want to go beyond simple analytics and business intelligence, into the AI/ML realm.</p><h4>Ataccama</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*QrQBWBRKhp5NIakv" /><figcaption>Image by author</figcaption></figure><ul><li>Pros: high-quality UI, integrated data quality, SaaS</li><li>Cons: custom ingestion protocol, lack of ML entities</li></ul><p><a href="https://www.ataccama.com/">Ataccama</a> is an AI-powered enterprise platform for data and metadata that features components for data quality, master data management, and data integration. The UI of the platform is good, and it enables fast analytical teams, highly regulated governance teams, and technical data teams to easily handle their data assets. One great perk of Ataccama is that it was designed with a strong focus on data quality. This is crucial for agile, data-driven organizations.</p><p>The downside is that Ataccama locks users in, because it requires a custom ingestion protocol. Though the situation is changing, it also lacks certain entities for machine learning.</p><h4>Monte Carlo</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*FMoOc2moWsvVbIDU" /><figcaption>Image by author</figcaption></figure><ul><li>Pros: decent UI, data observability features</li><li>Cons: custom ingestion protocol, lack of ML entities, focus on data observability as opposed to data discovery</li></ul><p><a href="https://www.montecarlodata.com/">Monte Carlo</a> is a data platform that helps data teams resolve the problem of data downtime, enabling them to handle their dashboards more efficiently, train more accurate ML models faster, and drive analytics operations. The platform is extensively focused on machine learning-powered data observability, which gives data teams an advanced capability to look into data and identify potential issues at scale. This gives Monte Carlo a considerable advantage over its competitors in data observability.</p><p>Like many other proprietary data catalog solutions, Monte Carlo locks the users into its own data ingestion protocol. It also lacks certain ML entities, which can be a deal-breaker for many organizations today.</p><h3>Conclusion</h3><p>Data catalogs are essential components of digital, data, and AI transformations. They organize all your data in one place and give you the tools to handle data more efficiently and on a larger scale. Data discovery, data observability, data governance, and other data-related activities can be improved and streamlined by incorporating a data catalog in your workflow.</p><p>I have explored almost 30 different data catalog vendors, and we have selected several top picks that you can use to compare and contrast different options. Some of them are open-source, some are proprietary, so you can pick and choose to find a solution that works for your team, organization, or company.</p><p>If you have any questions or suggestions, kindly share your feedback in the comment section, or reach out to me directly!</p><p><em>DISCLAIMER</em><br>This article offers my perspective on data catalog solutions. I did my best to be objective, but the community is sure to have something to add. Open discussion is always welcome and appreciated.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a265a4b3c0c3" width="1" height="1" alt=""><hr><p><a href="https://medium.com/provectus/finding-the-right-data-catalog-solution-a265a4b3c0c3">Finding the Right Data Catalog Solution</a> was originally published in <a href="https://medium.com/provectus">Provectus</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>